You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/03/23 16:40:42 UTC
[GitHub] [airflow] potiuk opened a new pull request #7841: Requirements now
depend on python version
potiuk opened a new pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841
---
Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397602652
##########
File path: CONTRIBUTING.rst
##########
@@ -341,31 +341,59 @@ be open to allow several different libraries with the same requirements to be in
The problem is that Apache Airflow is a bit of both - application to install and library to be used when
you are developing your own operators and DAGs.
-This - seemingly unsolvable - puzzle is solved as follows:
+This - seemingly unsolvable - puzzle is solved by having pinned requirement files. Those are available
+as of airflow 1.10.10.
-* by default when you install ``apache-airflow`` package - the dependencies are as open as possible while
- still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
- install in case a direct or transitive dependency is released that breaks the installation. In such case
- when installing ``apache-airflow``, you might need to provide additional constraints (for
- example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
+Pinned requirement files
+------------------------
-* we have ``requirements.txt`` file generated automatically based on the set of all latest working
- and tested requirement versions. You can also use that file as a constraints file when installing
- apache airflow - either from the sources ``pip install -e . --constraint requirements.txt`` or
- from the pypi package ``pip install apache-airflow --constraint requirements.txt``. Note that
- this will also work with extras for example ``pip install .[gcp] --constraint requirements.txt`` or
- ``pip install apache-airflow[gcp] --constraint requirements.txt``
+By default when you install ``apache-airflow`` package - the dependencies are as open as possible while
+still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
+install in case a direct or transitive dependency is released that breaks the installation. In such case
+when installing ``apache-airflow``, you might need to provide additional constraints (for
+example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
-The ``requirements.txt`` file should be updated automatically via pre-commit whenever you update dependencies
-It reflects the current set of dependencies installed in the CI image of Apache Airflow.
-The same set of requirements will be used to produce the production image.
+However we now havehave ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file generated
+automatically and committd in the requirements folder based on the set of all latest working and tested
+requirement versions. Those requirement.txt files can be used as constraints file when installing apache
+airflow - either from the sources ``pip install -e . --constraint requirements/requirements-python3.7.txt``
+or from the pypi package ``pip install apache-airflow --constraint requirements/requirements-python3.7.txt``.
+This work also with extras - ``pip install .[gcp] --constraint requirements/requirements-python3.7.txt``.
+It is also possible to use constraints directly from github (note that it only works starting from
+airflow 1.10.10 version:
-If you do not use pre-commits and the CI builds fails / you need to regenerate it, you can do it manually:
-``pre-commit run generate-requirements --all-files`` or via script
-``./scripts/ci/ci_generate_requirements.sh``.
-This will try to regenerate the requirements.txt file with the latest requirements matching
-the setup.py constraints.
+.. code-block:: bash
+
+ pip install apache-airflow[gcp]==1.10.10 \
+ --constraint https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt
+
+There are different set of fixed requirements for different python major/minor versions and you should
+use the right python version file for the right python version.
+
+The ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file MUST be regenerated every time after setup.py
Review comment:
Fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397539204
##########
File path: CONTRIBUTING.rst
##########
@@ -341,31 +341,59 @@ be open to allow several different libraries with the same requirements to be in
The problem is that Apache Airflow is a bit of both - application to install and library to be used when
you are developing your own operators and DAGs.
-This - seemingly unsolvable - puzzle is solved as follows:
+This - seemingly unsolvable - puzzle is solved by having pinned requirement files. Those are available
+as of airflow 1.10.10.
-* by default when you install ``apache-airflow`` package - the dependencies are as open as possible while
- still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
- install in case a direct or transitive dependency is released that breaks the installation. In such case
- when installing ``apache-airflow``, you might need to provide additional constraints (for
- example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
+Pinned requirement files
+------------------------
-* we have ``requirements.txt`` file generated automatically based on the set of all latest working
- and tested requirement versions. You can also use that file as a constraints file when installing
- apache airflow - either from the sources ``pip install -e . --constraint requirements.txt`` or
- from the pypi package ``pip install apache-airflow --constraint requirements.txt``. Note that
- this will also work with extras for example ``pip install .[gcp] --constraint requirements.txt`` or
- ``pip install apache-airflow[gcp] --constraint requirements.txt``
+By default when you install ``apache-airflow`` package - the dependencies are as open as possible while
+still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
+install in case a direct or transitive dependency is released that breaks the installation. In such case
+when installing ``apache-airflow``, you might need to provide additional constraints (for
+example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
-The ``requirements.txt`` file should be updated automatically via pre-commit whenever you update dependencies
-It reflects the current set of dependencies installed in the CI image of Apache Airflow.
-The same set of requirements will be used to produce the production image.
+However we now havehave ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file generated
+automatically and committd in the requirements folder based on the set of all latest working and tested
+requirement versions. Those requirement.txt files can be used as constraints file when installing apache
+airflow - either from the sources ``pip install -e . --constraint requirements/requirements-python3.7.txt``
+or from the pypi package ``pip install apache-airflow --constraint requirements/requirements-python3.7.txt``.
+This work also with extras - ``pip install .[gcp] --constraint requirements/requirements-python3.7.txt``.
+It is also possible to use constraints directly from github (note that it only works starting from
+airflow 1.10.10 version:
-If you do not use pre-commits and the CI builds fails / you need to regenerate it, you can do it manually:
-``pre-commit run generate-requirements --all-files`` or via script
-``./scripts/ci/ci_generate_requirements.sh``.
-This will try to regenerate the requirements.txt file with the latest requirements matching
-the setup.py constraints.
+.. code-block:: bash
+
+ pip install apache-airflow[gcp]==1.10.10 \
+ --constraint https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt
+
+There are different set of fixed requirements for different python major/minor versions and you should
+use the right python version file for the right python version.
+
+The ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file MUST be regenerated every time after setup.py
+is updated. This is checked automatically in Travis CI build there are separate jobs for each
Review comment:
```suggestion
is updated. This is checked automatically in Travis CI build. There are separate jobs for each
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397601320
##########
File path: CONTRIBUTING.rst
##########
@@ -341,31 +341,59 @@ be open to allow several different libraries with the same requirements to be in
The problem is that Apache Airflow is a bit of both - application to install and library to be used when
you are developing your own operators and DAGs.
-This - seemingly unsolvable - puzzle is solved as follows:
+This - seemingly unsolvable - puzzle is solved by having pinned requirement files. Those are available
+as of airflow 1.10.10.
-* by default when you install ``apache-airflow`` package - the dependencies are as open as possible while
- still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
- install in case a direct or transitive dependency is released that breaks the installation. In such case
- when installing ``apache-airflow``, you might need to provide additional constraints (for
- example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
+Pinned requirement files
+------------------------
-* we have ``requirements.txt`` file generated automatically based on the set of all latest working
- and tested requirement versions. You can also use that file as a constraints file when installing
- apache airflow - either from the sources ``pip install -e . --constraint requirements.txt`` or
- from the pypi package ``pip install apache-airflow --constraint requirements.txt``. Note that
- this will also work with extras for example ``pip install .[gcp] --constraint requirements.txt`` or
- ``pip install apache-airflow[gcp] --constraint requirements.txt``
+By default when you install ``apache-airflow`` package - the dependencies are as open as possible while
+still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
Review comment:
added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7841: Requirements
now depend on python version
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#issuecomment-603512068
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=h1) Report
> Merging [#7841](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/1982c3fdca1f04cfc41fc5b5e285d8f01c6b76ab&el=desc) will **decrease** coverage by `0.52%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7841 +/- ##
==========================================
- Coverage 86.97% 86.45% -0.53%
==========================================
Files 927 927
Lines 44963 44963
==========================================
- Hits 39108 38874 -234
- Misses 5855 6089 +234
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.25% <0.00%> (-72.50%)` | :arrow_down: |
| [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0.00%> (-64.16%)` | :arrow_down: |
| [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `47.82% <0.00%> (-52.18%)` | :arrow_down: |
| [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
| [airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==) | `53.33% <0.00%> (-46.67%)` | :arrow_down: |
| [airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5) | `55.00% <0.00%> (-45.00%)` | :arrow_down: |
| [airflow/providers/redis/sensors/redis\_key.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvc2Vuc29ycy9yZWRpc19rZXkucHk=) | `61.53% <0.00%> (-38.47%)` | :arrow_down: |
| [airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5) | `50.67% <0.00%> (-37.84%)` | :arrow_down: |
| [...roviders/google/cloud/operators/postgres\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9wb3N0Z3Jlc190b19nY3MucHk=) | `54.28% <0.00%> (-31.43%)` | :arrow_down: |
| [airflow/providers/postgres/hooks/postgres.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvaG9va3MvcG9zdGdyZXMucHk=) | `77.77% <0.00%> (-16.67%)` | :arrow_down: |
| ... and [7 more](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=footer). Last update [1982c3f...c1660bd](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397539036
##########
File path: CONTRIBUTING.rst
##########
@@ -341,31 +341,59 @@ be open to allow several different libraries with the same requirements to be in
The problem is that Apache Airflow is a bit of both - application to install and library to be used when
you are developing your own operators and DAGs.
-This - seemingly unsolvable - puzzle is solved as follows:
+This - seemingly unsolvable - puzzle is solved by having pinned requirement files. Those are available
+as of airflow 1.10.10.
-* by default when you install ``apache-airflow`` package - the dependencies are as open as possible while
- still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
- install in case a direct or transitive dependency is released that breaks the installation. In such case
- when installing ``apache-airflow``, you might need to provide additional constraints (for
- example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
+Pinned requirement files
+------------------------
-* we have ``requirements.txt`` file generated automatically based on the set of all latest working
- and tested requirement versions. You can also use that file as a constraints file when installing
- apache airflow - either from the sources ``pip install -e . --constraint requirements.txt`` or
- from the pypi package ``pip install apache-airflow --constraint requirements.txt``. Note that
- this will also work with extras for example ``pip install .[gcp] --constraint requirements.txt`` or
- ``pip install apache-airflow[gcp] --constraint requirements.txt``
+By default when you install ``apache-airflow`` package - the dependencies are as open as possible while
+still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
+install in case a direct or transitive dependency is released that breaks the installation. In such case
+when installing ``apache-airflow``, you might need to provide additional constraints (for
+example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
-The ``requirements.txt`` file should be updated automatically via pre-commit whenever you update dependencies
-It reflects the current set of dependencies installed in the CI image of Apache Airflow.
-The same set of requirements will be used to produce the production image.
+However we now havehave ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file generated
+automatically and committd in the requirements folder based on the set of all latest working and tested
+requirement versions. Those requirement.txt files can be used as constraints file when installing apache
+airflow - either from the sources ``pip install -e . --constraint requirements/requirements-python3.7.txt``
+or from the pypi package ``pip install apache-airflow --constraint requirements/requirements-python3.7.txt``.
+This work also with extras - ``pip install .[gcp] --constraint requirements/requirements-python3.7.txt``.
+It is also possible to use constraints directly from github (note that it only works starting from
+airflow 1.10.10 version:
-If you do not use pre-commits and the CI builds fails / you need to regenerate it, you can do it manually:
-``pre-commit run generate-requirements --all-files`` or via script
-``./scripts/ci/ci_generate_requirements.sh``.
-This will try to regenerate the requirements.txt file with the latest requirements matching
-the setup.py constraints.
+.. code-block:: bash
+
+ pip install apache-airflow[gcp]==1.10.10 \
+ --constraint https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt
+
+There are different set of fixed requirements for different python major/minor versions and you should
+use the right python version file for the right python version.
Review comment:
```suggestion
use the right requirements file for the right python version.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397541486
##########
File path: scripts/ci/docker-compose/base.yml
##########
@@ -59,6 +59,8 @@ services:
- DB_RESET
- ENABLED_SYSTEMS
- RUN_SYSTEM_TESTS
+ - DEFAULT_PYTHON_SHORT_VERSION
+ - PYTHON_SHORT_VERSION
Review comment:
Do we need both?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk edited a comment on issue #7841: Requirements now
depend on python version
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#issuecomment-605003271
OK. fixed separate requirements for different python versions is green in CI @kaxil @mik-laj -> do you want to take a look before I merge (@turbaszek ) ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397538278
##########
File path: CONTRIBUTING.rst
##########
@@ -341,31 +341,59 @@ be open to allow several different libraries with the same requirements to be in
The problem is that Apache Airflow is a bit of both - application to install and library to be used when
you are developing your own operators and DAGs.
-This - seemingly unsolvable - puzzle is solved as follows:
+This - seemingly unsolvable - puzzle is solved by having pinned requirement files. Those are available
+as of airflow 1.10.10.
-* by default when you install ``apache-airflow`` package - the dependencies are as open as possible while
- still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
- install in case a direct or transitive dependency is released that breaks the installation. In such case
- when installing ``apache-airflow``, you might need to provide additional constraints (for
- example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
+Pinned requirement files
+------------------------
-* we have ``requirements.txt`` file generated automatically based on the set of all latest working
- and tested requirement versions. You can also use that file as a constraints file when installing
- apache airflow - either from the sources ``pip install -e . --constraint requirements.txt`` or
- from the pypi package ``pip install apache-airflow --constraint requirements.txt``. Note that
- this will also work with extras for example ``pip install .[gcp] --constraint requirements.txt`` or
- ``pip install apache-airflow[gcp] --constraint requirements.txt``
+By default when you install ``apache-airflow`` package - the dependencies are as open as possible while
+still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
+install in case a direct or transitive dependency is released that breaks the installation. In such case
+when installing ``apache-airflow``, you might need to provide additional constraints (for
+example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
-The ``requirements.txt`` file should be updated automatically via pre-commit whenever you update dependencies
-It reflects the current set of dependencies installed in the CI image of Apache Airflow.
-The same set of requirements will be used to produce the production image.
+However we now havehave ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file generated
Review comment:
```suggestion
However, we now have ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file generated
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397602871
##########
File path: CONTRIBUTING.rst
##########
@@ -341,31 +341,59 @@ be open to allow several different libraries with the same requirements to be in
The problem is that Apache Airflow is a bit of both - application to install and library to be used when
you are developing your own operators and DAGs.
-This - seemingly unsolvable - puzzle is solved as follows:
+This - seemingly unsolvable - puzzle is solved by having pinned requirement files. Those are available
+as of airflow 1.10.10.
-* by default when you install ``apache-airflow`` package - the dependencies are as open as possible while
- still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
- install in case a direct or transitive dependency is released that breaks the installation. In such case
- when installing ``apache-airflow``, you might need to provide additional constraints (for
- example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
+Pinned requirement files
+------------------------
-* we have ``requirements.txt`` file generated automatically based on the set of all latest working
- and tested requirement versions. You can also use that file as a constraints file when installing
- apache airflow - either from the sources ``pip install -e . --constraint requirements.txt`` or
- from the pypi package ``pip install apache-airflow --constraint requirements.txt``. Note that
- this will also work with extras for example ``pip install .[gcp] --constraint requirements.txt`` or
- ``pip install apache-airflow[gcp] --constraint requirements.txt``
+By default when you install ``apache-airflow`` package - the dependencies are as open as possible while
+still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
+install in case a direct or transitive dependency is released that breaks the installation. In such case
+when installing ``apache-airflow``, you might need to provide additional constraints (for
+example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
-The ``requirements.txt`` file should be updated automatically via pre-commit whenever you update dependencies
-It reflects the current set of dependencies installed in the CI image of Apache Airflow.
-The same set of requirements will be used to produce the production image.
+However we now havehave ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file generated
+automatically and committd in the requirements folder based on the set of all latest working and tested
+requirement versions. Those requirement.txt files can be used as constraints file when installing apache
+airflow - either from the sources ``pip install -e . --constraint requirements/requirements-python3.7.txt``
+or from the pypi package ``pip install apache-airflow --constraint requirements/requirements-python3.7.txt``.
+This work also with extras - ``pip install .[gcp] --constraint requirements/requirements-python3.7.txt``.
+It is also possible to use constraints directly from github (note that it only works starting from
+airflow 1.10.10 version:
-If you do not use pre-commits and the CI builds fails / you need to regenerate it, you can do it manually:
-``pre-commit run generate-requirements --all-files`` or via script
-``./scripts/ci/ci_generate_requirements.sh``.
-This will try to regenerate the requirements.txt file with the latest requirements matching
-the setup.py constraints.
+.. code-block:: bash
+
+ pip install apache-airflow[gcp]==1.10.10 \
+ --constraint https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt
+
+There are different set of fixed requirements for different python major/minor versions and you should
+use the right python version file for the right python version.
+
+The ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file MUST be regenerated every time after setup.py
+is updated. This is checked automatically in Travis CI build there are separate jobs for each
+python version that checks if the requirements should be updated.
+
+If they are not you should regenerate the requirements locally using one of the methods described below.
+
+Generating requirement files
+----------------------------
+
+This shuould be done every time after you modify setup.py file. You can generate requirement files
Review comment:
Fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7841: Requirements now depend
on python version
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#issuecomment-603070770
BTW. I am going to re-write some parts of it. I came to the conclusion that generating requirements is too big of a hassle for pre-commits (especially that we need them in different versions) so I will do a separate breeze command (and script for those not using Breeze) so that whenever you modify setup.py and new requirements are needed you will be notified via CI rather than pre-commit. I think it's much less impact on the development workflow.
Bear with me :).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397538762
##########
File path: CONTRIBUTING.rst
##########
@@ -341,31 +341,59 @@ be open to allow several different libraries with the same requirements to be in
The problem is that Apache Airflow is a bit of both - application to install and library to be used when
you are developing your own operators and DAGs.
-This - seemingly unsolvable - puzzle is solved as follows:
+This - seemingly unsolvable - puzzle is solved by having pinned requirement files. Those are available
+as of airflow 1.10.10.
-* by default when you install ``apache-airflow`` package - the dependencies are as open as possible while
- still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
- install in case a direct or transitive dependency is released that breaks the installation. In such case
- when installing ``apache-airflow``, you might need to provide additional constraints (for
- example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
+Pinned requirement files
+------------------------
-* we have ``requirements.txt`` file generated automatically based on the set of all latest working
- and tested requirement versions. You can also use that file as a constraints file when installing
- apache airflow - either from the sources ``pip install -e . --constraint requirements.txt`` or
- from the pypi package ``pip install apache-airflow --constraint requirements.txt``. Note that
- this will also work with extras for example ``pip install .[gcp] --constraint requirements.txt`` or
- ``pip install apache-airflow[gcp] --constraint requirements.txt``
+By default when you install ``apache-airflow`` package - the dependencies are as open as possible while
+still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
+install in case a direct or transitive dependency is released that breaks the installation. In such case
+when installing ``apache-airflow``, you might need to provide additional constraints (for
+example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
-The ``requirements.txt`` file should be updated automatically via pre-commit whenever you update dependencies
-It reflects the current set of dependencies installed in the CI image of Apache Airflow.
-The same set of requirements will be used to produce the production image.
+However we now havehave ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file generated
+automatically and committd in the requirements folder based on the set of all latest working and tested
+requirement versions. Those requirement.txt files can be used as constraints file when installing apache
+airflow - either from the sources ``pip install -e . --constraint requirements/requirements-python3.7.txt``
+or from the pypi package ``pip install apache-airflow --constraint requirements/requirements-python3.7.txt``.
+This work also with extras - ``pip install .[gcp] --constraint requirements/requirements-python3.7.txt``.
Review comment:
```suggestion
This also works with extras - ``pip install .[gcp] --constraint requirements/requirements-python3.7.txt``.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7841: Requirements now depend
on python version
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#issuecomment-602719201
IT turned out that requirements are slightly different for different versions of python (especially in 1.10) so I had to add different version of requirements. At the same time I removed -pinned version. It's not really needed we should be able to easily install airflow now with one command:
```
pip install apache-airflow[gcp]==1.10.10 \
--constraint https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397603326
##########
File path: scripts/ci/_utils.sh
##########
@@ -530,7 +531,7 @@ function confirm_image_rebuild() {
# Make sure to use /dev/tty first rather than stdin/stdout when available - this way confirm
# will works also in case of pre-commits (git does not pass stdin/stdout to pre-commit hooks)
# shellcheck disable=SC2094
- "${AIRFLOW_SOURCES}/confirm" "${ACTION} image ${THE_IMAGE_TYPE} (might take some time)" \
+ "${AIRFLOW_SOURCES}/confirm" "${ACTION} image ${THE_IMAGE_TYPE}-${PYTHON_SHORT_VERSION}" \
Review comment:
Yep.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r399147202
##########
File path: .travis.yml
##########
@@ -41,24 +41,24 @@ jobs:
stage: pre-test
script: ./scripts/ci/ci_run_all_static_checks.sh
env: >-
- PYTHON_VERSION=3.6
+ PYTHON_MAJOR_MINOR_VERSION=3.6
Review comment:
It's not strange. It's MAJOR+MINOR version. PYTHON_VERSION is by default set in this image to 3.6.9 (so full PYTHON_VERSION) and what we really need here just MAJOR/MINOR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397602779
##########
File path: CONTRIBUTING.rst
##########
@@ -341,31 +341,59 @@ be open to allow several different libraries with the same requirements to be in
The problem is that Apache Airflow is a bit of both - application to install and library to be used when
you are developing your own operators and DAGs.
-This - seemingly unsolvable - puzzle is solved as follows:
+This - seemingly unsolvable - puzzle is solved by having pinned requirement files. Those are available
+as of airflow 1.10.10.
-* by default when you install ``apache-airflow`` package - the dependencies are as open as possible while
- still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
- install in case a direct or transitive dependency is released that breaks the installation. In such case
- when installing ``apache-airflow``, you might need to provide additional constraints (for
- example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
+Pinned requirement files
+------------------------
-* we have ``requirements.txt`` file generated automatically based on the set of all latest working
- and tested requirement versions. You can also use that file as a constraints file when installing
- apache airflow - either from the sources ``pip install -e . --constraint requirements.txt`` or
- from the pypi package ``pip install apache-airflow --constraint requirements.txt``. Note that
- this will also work with extras for example ``pip install .[gcp] --constraint requirements.txt`` or
- ``pip install apache-airflow[gcp] --constraint requirements.txt``
+By default when you install ``apache-airflow`` package - the dependencies are as open as possible while
+still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
+install in case a direct or transitive dependency is released that breaks the installation. In such case
+when installing ``apache-airflow``, you might need to provide additional constraints (for
+example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
-The ``requirements.txt`` file should be updated automatically via pre-commit whenever you update dependencies
-It reflects the current set of dependencies installed in the CI image of Apache Airflow.
-The same set of requirements will be used to produce the production image.
+However we now havehave ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file generated
+automatically and committd in the requirements folder based on the set of all latest working and tested
+requirement versions. Those requirement.txt files can be used as constraints file when installing apache
+airflow - either from the sources ``pip install -e . --constraint requirements/requirements-python3.7.txt``
+or from the pypi package ``pip install apache-airflow --constraint requirements/requirements-python3.7.txt``.
+This work also with extras - ``pip install .[gcp] --constraint requirements/requirements-python3.7.txt``.
+It is also possible to use constraints directly from github (note that it only works starting from
+airflow 1.10.10 version:
-If you do not use pre-commits and the CI builds fails / you need to regenerate it, you can do it manually:
-``pre-commit run generate-requirements --all-files`` or via script
-``./scripts/ci/ci_generate_requirements.sh``.
-This will try to regenerate the requirements.txt file with the latest requirements matching
-the setup.py constraints.
+.. code-block:: bash
+
+ pip install apache-airflow[gcp]==1.10.10 \
+ --constraint https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt
+
+There are different set of fixed requirements for different python major/minor versions and you should
+use the right python version file for the right python version.
+
+The ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file MUST be regenerated every time after setup.py
+is updated. This is checked automatically in Travis CI build there are separate jobs for each
Review comment:
Fixed,
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397603531
##########
File path: scripts/ci/docker-compose/base.yml
##########
@@ -59,6 +59,8 @@ services:
- DB_RESET
- ENABLED_SYSTEMS
- RUN_SYSTEM_TESTS
+ - DEFAULT_PYTHON_SHORT_VERSION
+ - PYTHON_SHORT_VERSION
Review comment:
Not any more :). It was only needed to generate requirements.txt file in the main repo but I removed it - it was nice for IDE's but very redundant and confusing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397539382
##########
File path: CONTRIBUTING.rst
##########
@@ -341,31 +341,59 @@ be open to allow several different libraries with the same requirements to be in
The problem is that Apache Airflow is a bit of both - application to install and library to be used when
you are developing your own operators and DAGs.
-This - seemingly unsolvable - puzzle is solved as follows:
+This - seemingly unsolvable - puzzle is solved by having pinned requirement files. Those are available
+as of airflow 1.10.10.
-* by default when you install ``apache-airflow`` package - the dependencies are as open as possible while
- still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
- install in case a direct or transitive dependency is released that breaks the installation. In such case
- when installing ``apache-airflow``, you might need to provide additional constraints (for
- example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
+Pinned requirement files
+------------------------
-* we have ``requirements.txt`` file generated automatically based on the set of all latest working
- and tested requirement versions. You can also use that file as a constraints file when installing
- apache airflow - either from the sources ``pip install -e . --constraint requirements.txt`` or
- from the pypi package ``pip install apache-airflow --constraint requirements.txt``. Note that
- this will also work with extras for example ``pip install .[gcp] --constraint requirements.txt`` or
- ``pip install apache-airflow[gcp] --constraint requirements.txt``
+By default when you install ``apache-airflow`` package - the dependencies are as open as possible while
+still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
+install in case a direct or transitive dependency is released that breaks the installation. In such case
+when installing ``apache-airflow``, you might need to provide additional constraints (for
+example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
-The ``requirements.txt`` file should be updated automatically via pre-commit whenever you update dependencies
-It reflects the current set of dependencies installed in the CI image of Apache Airflow.
-The same set of requirements will be used to produce the production image.
+However we now havehave ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file generated
+automatically and committd in the requirements folder based on the set of all latest working and tested
+requirement versions. Those requirement.txt files can be used as constraints file when installing apache
+airflow - either from the sources ``pip install -e . --constraint requirements/requirements-python3.7.txt``
+or from the pypi package ``pip install apache-airflow --constraint requirements/requirements-python3.7.txt``.
+This work also with extras - ``pip install .[gcp] --constraint requirements/requirements-python3.7.txt``.
+It is also possible to use constraints directly from github (note that it only works starting from
+airflow 1.10.10 version:
-If you do not use pre-commits and the CI builds fails / you need to regenerate it, you can do it manually:
-``pre-commit run generate-requirements --all-files`` or via script
-``./scripts/ci/ci_generate_requirements.sh``.
-This will try to regenerate the requirements.txt file with the latest requirements matching
-the setup.py constraints.
+.. code-block:: bash
+
+ pip install apache-airflow[gcp]==1.10.10 \
+ --constraint https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt
+
+There are different set of fixed requirements for different python major/minor versions and you should
+use the right python version file for the right python version.
+
+The ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file MUST be regenerated every time after setup.py
+is updated. This is checked automatically in Travis CI build there are separate jobs for each
+python version that checks if the requirements should be updated.
+
+If they are not you should regenerate the requirements locally using one of the methods described below.
+
+Generating requirement files
+----------------------------
+
+This shuould be done every time after you modify setup.py file. You can generate requirement files
Review comment:
```suggestion
This should be done every time after you modify setup.py file. You can generate requirement files
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397538872
##########
File path: CONTRIBUTING.rst
##########
@@ -341,31 +341,59 @@ be open to allow several different libraries with the same requirements to be in
The problem is that Apache Airflow is a bit of both - application to install and library to be used when
you are developing your own operators and DAGs.
-This - seemingly unsolvable - puzzle is solved as follows:
+This - seemingly unsolvable - puzzle is solved by having pinned requirement files. Those are available
+as of airflow 1.10.10.
-* by default when you install ``apache-airflow`` package - the dependencies are as open as possible while
- still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
- install in case a direct or transitive dependency is released that breaks the installation. In such case
- when installing ``apache-airflow``, you might need to provide additional constraints (for
- example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
+Pinned requirement files
+------------------------
-* we have ``requirements.txt`` file generated automatically based on the set of all latest working
- and tested requirement versions. You can also use that file as a constraints file when installing
- apache airflow - either from the sources ``pip install -e . --constraint requirements.txt`` or
- from the pypi package ``pip install apache-airflow --constraint requirements.txt``. Note that
- this will also work with extras for example ``pip install .[gcp] --constraint requirements.txt`` or
- ``pip install apache-airflow[gcp] --constraint requirements.txt``
+By default when you install ``apache-airflow`` package - the dependencies are as open as possible while
+still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
+install in case a direct or transitive dependency is released that breaks the installation. In such case
+when installing ``apache-airflow``, you might need to provide additional constraints (for
+example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
-The ``requirements.txt`` file should be updated automatically via pre-commit whenever you update dependencies
-It reflects the current set of dependencies installed in the CI image of Apache Airflow.
-The same set of requirements will be used to produce the production image.
+However we now havehave ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file generated
+automatically and committd in the requirements folder based on the set of all latest working and tested
+requirement versions. Those requirement.txt files can be used as constraints file when installing apache
+airflow - either from the sources ``pip install -e . --constraint requirements/requirements-python3.7.txt``
+or from the pypi package ``pip install apache-airflow --constraint requirements/requirements-python3.7.txt``.
+This work also with extras - ``pip install .[gcp] --constraint requirements/requirements-python3.7.txt``.
+It is also possible to use constraints directly from github (note that it only works starting from
+airflow 1.10.10 version:
Review comment:
```suggestion
Airflow 1.10.10 version:
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7841: Requirements now depend
on python version
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#issuecomment-605003271
OK. fixed separate requirements for different python versions is green in CI @kaxil @mik-laj -> do you want to take a look ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk merged pull request #7841: Requirements now depend
on python version
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7841: Requirements now depend
on python version
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#issuecomment-603441221
@kaxil @ashb @turbaszek @mik-laj @dimberman -> We have much nicer version now. Requirement generation is now checked only as travis build (separately for each python version) rather than pre-commit and you can esily manually generate the requirements with breeze (examples in doc). Also when you modify setup.py in the way that it will require regeneration of the reuqirements, the build in CI will fail, show you with nice, coloured diff what is added and will ask you to run `breeze generate-requirements --python x.y` as needed.
Pls review. That's the last thing to cherry-pick and test prod image in airflow 1.10.10
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7841: Requirements now
depend on python version
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#issuecomment-603512068
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=h1) Report
> Merging [#7841](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/1982c3fdca1f04cfc41fc5b5e285d8f01c6b76ab&el=desc) will **decrease** coverage by `1.01%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7841 +/- ##
==========================================
- Coverage 86.97% 85.95% -1.02%
==========================================
Files 927 927
Lines 44963 44963
==========================================
- Hits 39108 38650 -458
- Misses 5855 6313 +458
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.25% <0.00%> (-72.50%)` | :arrow_down: |
| [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0.00%> (-64.16%)` | :arrow_down: |
| [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `39.28% <0.00%> (-60.72%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
| [airflow/api/auth/backend/kerberos\_auth.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9hcGkvYXV0aC9iYWNrZW5kL2tlcmJlcm9zX2F1dGgucHk=) | `28.16% <0.00%> (-54.93%)` | :arrow_down: |
| [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `47.82% <0.00%> (-52.18%)` | :arrow_down: |
| [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: |
| [airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==) | `53.33% <0.00%> (-46.67%)` | :arrow_down: |
| [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `30.43% <0.00%> (-45.66%)` | :arrow_down: |
| ... and [17 more](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=footer). Last update [1982c3f...c1660bd](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397538375
##########
File path: CONTRIBUTING.rst
##########
@@ -341,31 +341,59 @@ be open to allow several different libraries with the same requirements to be in
The problem is that Apache Airflow is a bit of both - application to install and library to be used when
you are developing your own operators and DAGs.
-This - seemingly unsolvable - puzzle is solved as follows:
+This - seemingly unsolvable - puzzle is solved by having pinned requirement files. Those are available
+as of airflow 1.10.10.
-* by default when you install ``apache-airflow`` package - the dependencies are as open as possible while
- still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
- install in case a direct or transitive dependency is released that breaks the installation. In such case
- when installing ``apache-airflow``, you might need to provide additional constraints (for
- example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
+Pinned requirement files
+------------------------
-* we have ``requirements.txt`` file generated automatically based on the set of all latest working
- and tested requirement versions. You can also use that file as a constraints file when installing
- apache airflow - either from the sources ``pip install -e . --constraint requirements.txt`` or
- from the pypi package ``pip install apache-airflow --constraint requirements.txt``. Note that
- this will also work with extras for example ``pip install .[gcp] --constraint requirements.txt`` or
- ``pip install apache-airflow[gcp] --constraint requirements.txt``
+By default when you install ``apache-airflow`` package - the dependencies are as open as possible while
+still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
+install in case a direct or transitive dependency is released that breaks the installation. In such case
+when installing ``apache-airflow``, you might need to provide additional constraints (for
+example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
-The ``requirements.txt`` file should be updated automatically via pre-commit whenever you update dependencies
-It reflects the current set of dependencies installed in the CI image of Apache Airflow.
-The same set of requirements will be used to produce the production image.
+However we now havehave ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file generated
+automatically and committd in the requirements folder based on the set of all latest working and tested
Review comment:
```suggestion
automatically and committed in the requirements folder based on the set of all the latest working and tested
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7841: Requirements
now depend on python version
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#issuecomment-603512068
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=h1) Report
> Merging [#7841](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/3cc37b1fed388b5bfb9a153443fb1e60282df5db&el=desc) will **decrease** coverage by `22.25%`.
> The diff coverage is `63.04%`.
[](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7841 +/- ##
===========================================
- Coverage 86.97% 64.72% -22.26%
===========================================
Files 926 927 +1
Lines 44982 44966 -16
===========================================
- Hits 39123 29104 -10019
- Misses 5859 15862 +10003
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/executors/dask\_executor.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvZGFza19leGVjdXRvci5weQ==) | `25.75% <0.00%> (+19.87%)` | :arrow_up: |
| [airflow/operators/dummy\_operator.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZHVtbXlfb3BlcmF0b3IucHk=) | `88.88% <ø> (-11.12%)` | :arrow_down: |
| [airflow/operators/presto\_check\_operator.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcHJlc3RvX2NoZWNrX29wZXJhdG9yLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/providers/google/cloud/hooks/cloud\_sql.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2hvb2tzL2Nsb3VkX3NxbC5weQ==) | `67.25% <0.00%> (ø)` | |
| [.../google/cloud/utils/mlengine\_prediction\_summary.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL3V0aWxzL21sZW5naW5lX3ByZWRpY3Rpb25fc3VtbWFyeS5weQ==) | `0.00% <0.00%> (ø)` | |
| [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `25.78% <0.00%> (-50.49%)` | :arrow_down: |
| [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `14.26% <33.33%> (-76.62%)` | :arrow_down: |
| [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `12.92% <50.00%> (-79.20%)` | :arrow_down: |
| [airflow/providers/apache/hdfs/hooks/webhdfs.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hkZnMvaG9va3Mvd2ViaGRmcy5weQ==) | `86.66% <85.71%> (-3.13%)` | :arrow_down: |
| [airflow/secrets/base\_secrets.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZWNyZXRzL2Jhc2Vfc2VjcmV0cy5weQ==) | `94.11% <94.11%> (ø)` | |
| ... and [515 more](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=footer). Last update [6486abc...9979f94](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397539097
##########
File path: CONTRIBUTING.rst
##########
@@ -341,31 +341,59 @@ be open to allow several different libraries with the same requirements to be in
The problem is that Apache Airflow is a bit of both - application to install and library to be used when
you are developing your own operators and DAGs.
-This - seemingly unsolvable - puzzle is solved as follows:
+This - seemingly unsolvable - puzzle is solved by having pinned requirement files. Those are available
+as of airflow 1.10.10.
-* by default when you install ``apache-airflow`` package - the dependencies are as open as possible while
- still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
- install in case a direct or transitive dependency is released that breaks the installation. In such case
- when installing ``apache-airflow``, you might need to provide additional constraints (for
- example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
+Pinned requirement files
+------------------------
-* we have ``requirements.txt`` file generated automatically based on the set of all latest working
- and tested requirement versions. You can also use that file as a constraints file when installing
- apache airflow - either from the sources ``pip install -e . --constraint requirements.txt`` or
- from the pypi package ``pip install apache-airflow --constraint requirements.txt``. Note that
- this will also work with extras for example ``pip install .[gcp] --constraint requirements.txt`` or
- ``pip install apache-airflow[gcp] --constraint requirements.txt``
+By default when you install ``apache-airflow`` package - the dependencies are as open as possible while
+still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
+install in case a direct or transitive dependency is released that breaks the installation. In such case
+when installing ``apache-airflow``, you might need to provide additional constraints (for
+example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
-The ``requirements.txt`` file should be updated automatically via pre-commit whenever you update dependencies
-It reflects the current set of dependencies installed in the CI image of Apache Airflow.
-The same set of requirements will be used to produce the production image.
+However we now havehave ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file generated
+automatically and committd in the requirements folder based on the set of all latest working and tested
+requirement versions. Those requirement.txt files can be used as constraints file when installing apache
+airflow - either from the sources ``pip install -e . --constraint requirements/requirements-python3.7.txt``
+or from the pypi package ``pip install apache-airflow --constraint requirements/requirements-python3.7.txt``.
+This work also with extras - ``pip install .[gcp] --constraint requirements/requirements-python3.7.txt``.
+It is also possible to use constraints directly from github (note that it only works starting from
+airflow 1.10.10 version:
-If you do not use pre-commits and the CI builds fails / you need to regenerate it, you can do it manually:
-``pre-commit run generate-requirements --all-files`` or via script
-``./scripts/ci/ci_generate_requirements.sh``.
-This will try to regenerate the requirements.txt file with the latest requirements matching
-the setup.py constraints.
+.. code-block:: bash
+
+ pip install apache-airflow[gcp]==1.10.10 \
+ --constraint https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt
+
+There are different set of fixed requirements for different python major/minor versions and you should
+use the right python version file for the right python version.
+
+The ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file MUST be regenerated every time after setup.py
Review comment:
```suggestion
The ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file MUST be regenerated every time after ``setup.py``
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397539315
##########
File path: CONTRIBUTING.rst
##########
@@ -341,31 +341,59 @@ be open to allow several different libraries with the same requirements to be in
The problem is that Apache Airflow is a bit of both - application to install and library to be used when
you are developing your own operators and DAGs.
-This - seemingly unsolvable - puzzle is solved as follows:
+This - seemingly unsolvable - puzzle is solved by having pinned requirement files. Those are available
+as of airflow 1.10.10.
-* by default when you install ``apache-airflow`` package - the dependencies are as open as possible while
- still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
- install in case a direct or transitive dependency is released that breaks the installation. In such case
- when installing ``apache-airflow``, you might need to provide additional constraints (for
- example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
+Pinned requirement files
+------------------------
-* we have ``requirements.txt`` file generated automatically based on the set of all latest working
- and tested requirement versions. You can also use that file as a constraints file when installing
- apache airflow - either from the sources ``pip install -e . --constraint requirements.txt`` or
- from the pypi package ``pip install apache-airflow --constraint requirements.txt``. Note that
- this will also work with extras for example ``pip install .[gcp] --constraint requirements.txt`` or
- ``pip install apache-airflow[gcp] --constraint requirements.txt``
+By default when you install ``apache-airflow`` package - the dependencies are as open as possible while
+still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
+install in case a direct or transitive dependency is released that breaks the installation. In such case
+when installing ``apache-airflow``, you might need to provide additional constraints (for
+example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
-The ``requirements.txt`` file should be updated automatically via pre-commit whenever you update dependencies
-It reflects the current set of dependencies installed in the CI image of Apache Airflow.
-The same set of requirements will be used to produce the production image.
+However we now havehave ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file generated
+automatically and committd in the requirements folder based on the set of all latest working and tested
+requirement versions. Those requirement.txt files can be used as constraints file when installing apache
+airflow - either from the sources ``pip install -e . --constraint requirements/requirements-python3.7.txt``
+or from the pypi package ``pip install apache-airflow --constraint requirements/requirements-python3.7.txt``.
+This work also with extras - ``pip install .[gcp] --constraint requirements/requirements-python3.7.txt``.
+It is also possible to use constraints directly from github (note that it only works starting from
+airflow 1.10.10 version:
-If you do not use pre-commits and the CI builds fails / you need to regenerate it, you can do it manually:
-``pre-commit run generate-requirements --all-files`` or via script
-``./scripts/ci/ci_generate_requirements.sh``.
-This will try to regenerate the requirements.txt file with the latest requirements matching
-the setup.py constraints.
+.. code-block:: bash
+
+ pip install apache-airflow[gcp]==1.10.10 \
+ --constraint https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt
+
+There are different set of fixed requirements for different python major/minor versions and you should
+use the right python version file for the right python version.
+
+The ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file MUST be regenerated every time after setup.py
+is updated. This is checked automatically in Travis CI build there are separate jobs for each
+python version that checks if the requirements should be updated.
+
+If they are not you should regenerate the requirements locally using one of the methods described below.
Review comment:
```suggestion
If they are not updated, you should regenerate the requirements locally using one of the methods described below.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397538596
##########
File path: CONTRIBUTING.rst
##########
@@ -341,31 +341,59 @@ be open to allow several different libraries with the same requirements to be in
The problem is that Apache Airflow is a bit of both - application to install and library to be used when
you are developing your own operators and DAGs.
-This - seemingly unsolvable - puzzle is solved as follows:
+This - seemingly unsolvable - puzzle is solved by having pinned requirement files. Those are available
+as of airflow 1.10.10.
-* by default when you install ``apache-airflow`` package - the dependencies are as open as possible while
- still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
- install in case a direct or transitive dependency is released that breaks the installation. In such case
- when installing ``apache-airflow``, you might need to provide additional constraints (for
- example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
+Pinned requirement files
+------------------------
-* we have ``requirements.txt`` file generated automatically based on the set of all latest working
- and tested requirement versions. You can also use that file as a constraints file when installing
- apache airflow - either from the sources ``pip install -e . --constraint requirements.txt`` or
- from the pypi package ``pip install apache-airflow --constraint requirements.txt``. Note that
- this will also work with extras for example ``pip install .[gcp] --constraint requirements.txt`` or
- ``pip install apache-airflow[gcp] --constraint requirements.txt``
+By default when you install ``apache-airflow`` package - the dependencies are as open as possible while
+still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
+install in case a direct or transitive dependency is released that breaks the installation. In such case
+when installing ``apache-airflow``, you might need to provide additional constraints (for
+example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
-The ``requirements.txt`` file should be updated automatically via pre-commit whenever you update dependencies
-It reflects the current set of dependencies installed in the CI image of Apache Airflow.
-The same set of requirements will be used to produce the production image.
+However we now havehave ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file generated
+automatically and committd in the requirements folder based on the set of all latest working and tested
+requirement versions. Those requirement.txt files can be used as constraints file when installing apache
Review comment:
```suggestion
requirement versions. Those ``requirements.txt`` files can be used as constraints file when installing apache
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r396942564
##########
File path: .travis.yml
##########
@@ -41,24 +41,24 @@ jobs:
stage: pre-test
script: ./scripts/ci/ci_run_all_static_checks.sh
env: >-
- PYTHON_VERSION=3.6
+ PYTHON_SHORT_VERSION=3.6
Review comment:
Yes. PYTHON_VERSION is set by default by the python base image and it is the full version (PYTHON_VERSION=3.7.9). It got so confusing that I decided I finally have to separate those two.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397538133
##########
File path: CONTRIBUTING.rst
##########
@@ -341,31 +341,59 @@ be open to allow several different libraries with the same requirements to be in
The problem is that Apache Airflow is a bit of both - application to install and library to be used when
you are developing your own operators and DAGs.
-This - seemingly unsolvable - puzzle is solved as follows:
+This - seemingly unsolvable - puzzle is solved by having pinned requirement files. Those are available
+as of airflow 1.10.10.
-* by default when you install ``apache-airflow`` package - the dependencies are as open as possible while
- still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
- install in case a direct or transitive dependency is released that breaks the installation. In such case
- when installing ``apache-airflow``, you might need to provide additional constraints (for
- example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
+Pinned requirement files
+------------------------
-* we have ``requirements.txt`` file generated automatically based on the set of all latest working
- and tested requirement versions. You can also use that file as a constraints file when installing
- apache airflow - either from the sources ``pip install -e . --constraint requirements.txt`` or
- from the pypi package ``pip install apache-airflow --constraint requirements.txt``. Note that
- this will also work with extras for example ``pip install .[gcp] --constraint requirements.txt`` or
- ``pip install apache-airflow[gcp] --constraint requirements.txt``
+By default when you install ``apache-airflow`` package - the dependencies are as open as possible while
+still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
Review comment:
```suggestion
still allowing the apache-airflow package to install. This means that 'apache-airflow' package might fail to
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397537666
##########
File path: .travis.yml
##########
@@ -41,24 +41,24 @@ jobs:
stage: pre-test
script: ./scripts/ci/ci_run_all_static_checks.sh
env: >-
- PYTHON_VERSION=3.6
+ PYTHON_SHORT_VERSION=3.6
Review comment:
Sounds good
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7841: Requirements
now depend on python version
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#issuecomment-603512068
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=h1) Report
> Merging [#7841](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/1982c3fdca1f04cfc41fc5b5e285d8f01c6b76ab&el=desc) will **decrease** coverage by `0.75%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7841 +/- ##
==========================================
- Coverage 86.97% 86.22% -0.76%
==========================================
Files 927 927
Lines 44963 44963
==========================================
- Hits 39108 38769 -339
- Misses 5855 6194 +339
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.25% <0.00%> (-72.50%)` | :arrow_down: |
| [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0.00%> (-64.16%)` | :arrow_down: |
| [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `39.28% <0.00%> (-60.72%)` | :arrow_down: |
| [airflow/api/auth/backend/kerberos\_auth.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9hcGkvYXV0aC9iYWNrZW5kL2tlcmJlcm9zX2F1dGgucHk=) | `28.16% <0.00%> (-54.93%)` | :arrow_down: |
| [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `47.82% <0.00%> (-52.18%)` | :arrow_down: |
| [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
| [airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==) | `53.33% <0.00%> (-46.67%)` | :arrow_down: |
| [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `30.43% <0.00%> (-45.66%)` | :arrow_down: |
| [airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5) | `55.00% <0.00%> (-45.00%)` | :arrow_down: |
| [airflow/providers/redis/sensors/redis\_key.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvc2Vuc29ycy9yZWRpc19rZXkucHk=) | `61.53% <0.00%> (-38.47%)` | :arrow_down: |
| ... and [12 more](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=footer). Last update [1982c3f...c1660bd](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397538375
##########
File path: CONTRIBUTING.rst
##########
@@ -341,31 +341,59 @@ be open to allow several different libraries with the same requirements to be in
The problem is that Apache Airflow is a bit of both - application to install and library to be used when
you are developing your own operators and DAGs.
-This - seemingly unsolvable - puzzle is solved as follows:
+This - seemingly unsolvable - puzzle is solved by having pinned requirement files. Those are available
+as of airflow 1.10.10.
-* by default when you install ``apache-airflow`` package - the dependencies are as open as possible while
- still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
- install in case a direct or transitive dependency is released that breaks the installation. In such case
- when installing ``apache-airflow``, you might need to provide additional constraints (for
- example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
+Pinned requirement files
+------------------------
-* we have ``requirements.txt`` file generated automatically based on the set of all latest working
- and tested requirement versions. You can also use that file as a constraints file when installing
- apache airflow - either from the sources ``pip install -e . --constraint requirements.txt`` or
- from the pypi package ``pip install apache-airflow --constraint requirements.txt``. Note that
- this will also work with extras for example ``pip install .[gcp] --constraint requirements.txt`` or
- ``pip install apache-airflow[gcp] --constraint requirements.txt``
+By default when you install ``apache-airflow`` package - the dependencies are as open as possible while
+still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
+install in case a direct or transitive dependency is released that breaks the installation. In such case
+when installing ``apache-airflow``, you might need to provide additional constraints (for
+example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
-The ``requirements.txt`` file should be updated automatically via pre-commit whenever you update dependencies
-It reflects the current set of dependencies installed in the CI image of Apache Airflow.
-The same set of requirements will be used to produce the production image.
+However we now havehave ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file generated
+automatically and committd in the requirements folder based on the set of all latest working and tested
Review comment:
```suggestion
automatically and committed in the requirements folder based on the set of all latest working and tested
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397538396
##########
File path: .travis.yml
##########
@@ -41,24 +41,24 @@ jobs:
stage: pre-test
script: ./scripts/ci/ci_run_all_static_checks.sh
env: >-
- PYTHON_VERSION=3.6
+ PYTHON_SHORT_VERSION=3.6
Review comment:
What do you think about renaming to ``PYTHON_MINOR_VERSION``?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397601489
##########
File path: CONTRIBUTING.rst
##########
@@ -341,31 +341,59 @@ be open to allow several different libraries with the same requirements to be in
The problem is that Apache Airflow is a bit of both - application to install and library to be used when
you are developing your own operators and DAGs.
-This - seemingly unsolvable - puzzle is solved as follows:
+This - seemingly unsolvable - puzzle is solved by having pinned requirement files. Those are available
+as of airflow 1.10.10.
-* by default when you install ``apache-airflow`` package - the dependencies are as open as possible while
- still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
- install in case a direct or transitive dependency is released that breaks the installation. In such case
- when installing ``apache-airflow``, you might need to provide additional constraints (for
- example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
+Pinned requirement files
+------------------------
-* we have ``requirements.txt`` file generated automatically based on the set of all latest working
- and tested requirement versions. You can also use that file as a constraints file when installing
- apache airflow - either from the sources ``pip install -e . --constraint requirements.txt`` or
- from the pypi package ``pip install apache-airflow --constraint requirements.txt``. Note that
- this will also work with extras for example ``pip install .[gcp] --constraint requirements.txt`` or
- ``pip install apache-airflow[gcp] --constraint requirements.txt``
+By default when you install ``apache-airflow`` package - the dependencies are as open as possible while
+still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
+install in case a direct or transitive dependency is released that breaks the installation. In such case
+when installing ``apache-airflow``, you might need to provide additional constraints (for
+example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
-The ``requirements.txt`` file should be updated automatically via pre-commit whenever you update dependencies
-It reflects the current set of dependencies installed in the CI image of Apache Airflow.
-The same set of requirements will be used to produce the production image.
+However we now havehave ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file generated
+automatically and committd in the requirements folder based on the set of all latest working and tested
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397665648
##########
File path: .travis.yml
##########
@@ -41,24 +41,24 @@ jobs:
stage: pre-test
script: ./scripts/ci/ci_run_all_static_checks.sh
env: >-
- PYTHON_VERSION=3.6
+ PYTHON_MAJOR_MINOR_VERSION=3.6
Review comment:
Why such strange name?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r398188442
##########
File path: Dockerfile.ci
##########
@@ -40,14 +43,19 @@ ARG DEPENDENCIES_EPOCH_NUMBER="2"
ENV DEPENDENCIES_EPOCH_NUMBER=${DEPENDENCIES_EPOCH_NUMBER}
# Install curl and gnupg2 - needed to download nodejs in the next step
+# also netselect-apt is used to choose the best mirror
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
curl \
gnupg2 \
+ netselect-apt \
&& apt-get autoremove -yqq --purge \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
+# Choose faster mirror to download dependencies
+RUN netselect-apt -o /etc/apt/sources.list
Review comment:
+1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r399147338
##########
File path: .travis.yml
##########
@@ -41,24 +41,24 @@ jobs:
stage: pre-test
script: ./scripts/ci/ci_run_all_static_checks.sh
env: >-
- PYTHON_VERSION=3.6
+ PYTHON_MAJOR_MINOR_VERSION=3.6
Review comment:
Are you OK with it @turbaszek ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397601566
##########
File path: CONTRIBUTING.rst
##########
@@ -341,31 +341,59 @@ be open to allow several different libraries with the same requirements to be in
The problem is that Apache Airflow is a bit of both - application to install and library to be used when
you are developing your own operators and DAGs.
-This - seemingly unsolvable - puzzle is solved as follows:
+This - seemingly unsolvable - puzzle is solved by having pinned requirement files. Those are available
+as of airflow 1.10.10.
-* by default when you install ``apache-airflow`` package - the dependencies are as open as possible while
- still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
- install in case a direct or transitive dependency is released that breaks the installation. In such case
- when installing ``apache-airflow``, you might need to provide additional constraints (for
- example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
+Pinned requirement files
+------------------------
-* we have ``requirements.txt`` file generated automatically based on the set of all latest working
- and tested requirement versions. You can also use that file as a constraints file when installing
- apache airflow - either from the sources ``pip install -e . --constraint requirements.txt`` or
- from the pypi package ``pip install apache-airflow --constraint requirements.txt``. Note that
- this will also work with extras for example ``pip install .[gcp] --constraint requirements.txt`` or
- ``pip install apache-airflow[gcp] --constraint requirements.txt``
+By default when you install ``apache-airflow`` package - the dependencies are as open as possible while
+still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
+install in case a direct or transitive dependency is released that breaks the installation. In such case
+when installing ``apache-airflow``, you might need to provide additional constraints (for
+example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
-The ``requirements.txt`` file should be updated automatically via pre-commit whenever you update dependencies
-It reflects the current set of dependencies installed in the CI image of Apache Airflow.
-The same set of requirements will be used to produce the production image.
+However we now havehave ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file generated
Review comment:
Fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397540234
##########
File path: CONTRIBUTING.rst
##########
@@ -341,31 +341,59 @@ be open to allow several different libraries with the same requirements to be in
The problem is that Apache Airflow is a bit of both - application to install and library to be used when
you are developing your own operators and DAGs.
-This - seemingly unsolvable - puzzle is solved as follows:
+This - seemingly unsolvable - puzzle is solved by having pinned requirement files. Those are available
+as of airflow 1.10.10.
Review comment:
```suggestion
as of Airflow 1.10.10.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7841: Requirements
now depend on python version
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#issuecomment-603512068
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=h1) Report
> Merging [#7841](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/2a98a61f596fa463f921aa4203a3b2019e2ac3a2?src=pr&el=desc) will **decrease** coverage by `22.37%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7841 +/- ##
===========================================
- Coverage 87.1% 64.72% -22.38%
===========================================
Files 928 927 -1
Lines 45025 44966 -59
===========================================
- Hits 39217 29104 -10113
- Misses 5808 15862 +10054
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...low/contrib/operators/wasb\_delete\_blob\_operator.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy93YXNiX2RlbGV0ZV9ibG9iX29wZXJhdG9yLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/contrib/hooks/vertica\_hook.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3ZlcnRpY2FfaG9vay5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/contrib/sensors/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvX19pbml0X18ucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/hooks/mssql\_hook.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9tc3NxbF9ob29rLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [...viders/docker/example\_dags/example\_docker\_swarm.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZG9ja2VyL2V4YW1wbGVfZGFncy9leGFtcGxlX2RvY2tlcl9zd2FybS5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/hooks/webhdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy93ZWJoZGZzX2hvb2sucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/contrib/sensors/emr\_base\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvZW1yX2Jhc2Vfc2Vuc29yLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [...irflow/contrib/operators/slack\_webhook\_operator.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zbGFja193ZWJob29rX29wZXJhdG9yLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [...providers/google/cloud/example\_dags/example\_dlp.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2V4YW1wbGVfZGFncy9leGFtcGxlX2RscC5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [...cncf/kubernetes/example\_dags/example\_kubernetes.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL2V4YW1wbGVfZGFncy9leGFtcGxlX2t1YmVybmV0ZXMucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| ... and [501 more](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=footer). Last update [2a98a61...776f33c](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397601090
##########
File path: .travis.yml
##########
@@ -41,24 +41,24 @@ jobs:
stage: pre-test
script: ./scripts/ci/ci_run_all_static_checks.sh
env: >-
- PYTHON_VERSION=3.6
+ PYTHON_SHORT_VERSION=3.6
Review comment:
PYTHON_MAJOR_MINOR_VERSION is better
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397538278
##########
File path: CONTRIBUTING.rst
##########
@@ -341,31 +341,59 @@ be open to allow several different libraries with the same requirements to be in
The problem is that Apache Airflow is a bit of both - application to install and library to be used when
you are developing your own operators and DAGs.
-This - seemingly unsolvable - puzzle is solved as follows:
+This - seemingly unsolvable - puzzle is solved by having pinned requirement files. Those are available
+as of airflow 1.10.10.
-* by default when you install ``apache-airflow`` package - the dependencies are as open as possible while
- still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
- install in case a direct or transitive dependency is released that breaks the installation. In such case
- when installing ``apache-airflow``, you might need to provide additional constraints (for
- example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
+Pinned requirement files
+------------------------
-* we have ``requirements.txt`` file generated automatically based on the set of all latest working
- and tested requirement versions. You can also use that file as a constraints file when installing
- apache airflow - either from the sources ``pip install -e . --constraint requirements.txt`` or
- from the pypi package ``pip install apache-airflow --constraint requirements.txt``. Note that
- this will also work with extras for example ``pip install .[gcp] --constraint requirements.txt`` or
- ``pip install apache-airflow[gcp] --constraint requirements.txt``
+By default when you install ``apache-airflow`` package - the dependencies are as open as possible while
+still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
+install in case a direct or transitive dependency is released that breaks the installation. In such case
+when installing ``apache-airflow``, you might need to provide additional constraints (for
+example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
-The ``requirements.txt`` file should be updated automatically via pre-commit whenever you update dependencies
-It reflects the current set of dependencies installed in the CI image of Apache Airflow.
-The same set of requirements will be used to produce the production image.
+However we now havehave ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file generated
Review comment:
```suggestion
However we now have ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file generated
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397540533
##########
File path: scripts/ci/_utils.sh
##########
@@ -530,7 +531,7 @@ function confirm_image_rebuild() {
# Make sure to use /dev/tty first rather than stdin/stdout when available - this way confirm
# will works also in case of pre-commits (git does not pass stdin/stdout to pre-commit hooks)
# shellcheck disable=SC2094
- "${AIRFLOW_SOURCES}/confirm" "${ACTION} image ${THE_IMAGE_TYPE} (might take some time)" \
+ "${AIRFLOW_SOURCES}/confirm" "${ACTION} image ${THE_IMAGE_TYPE}-${PYTHON_SHORT_VERSION}" \
Review comment:
```suggestion
"${AIRFLOW_SOURCES}/confirm" "${ACTION} image ${THE_IMAGE_TYPE}-python ${PYTHON_SHORT_VERSION}" \
```
?? not sure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397602286
##########
File path: CONTRIBUTING.rst
##########
@@ -341,31 +341,59 @@ be open to allow several different libraries with the same requirements to be in
The problem is that Apache Airflow is a bit of both - application to install and library to be used when
you are developing your own operators and DAGs.
-This - seemingly unsolvable - puzzle is solved as follows:
+This - seemingly unsolvable - puzzle is solved by having pinned requirement files. Those are available
+as of airflow 1.10.10.
-* by default when you install ``apache-airflow`` package - the dependencies are as open as possible while
- still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
- install in case a direct or transitive dependency is released that breaks the installation. In such case
- when installing ``apache-airflow``, you might need to provide additional constraints (for
- example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
+Pinned requirement files
+------------------------
-* we have ``requirements.txt`` file generated automatically based on the set of all latest working
- and tested requirement versions. You can also use that file as a constraints file when installing
- apache airflow - either from the sources ``pip install -e . --constraint requirements.txt`` or
- from the pypi package ``pip install apache-airflow --constraint requirements.txt``. Note that
- this will also work with extras for example ``pip install .[gcp] --constraint requirements.txt`` or
- ``pip install apache-airflow[gcp] --constraint requirements.txt``
+By default when you install ``apache-airflow`` package - the dependencies are as open as possible while
+still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
+install in case a direct or transitive dependency is released that breaks the installation. In such case
+when installing ``apache-airflow``, you might need to provide additional constraints (for
+example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
-The ``requirements.txt`` file should be updated automatically via pre-commit whenever you update dependencies
-It reflects the current set of dependencies installed in the CI image of Apache Airflow.
-The same set of requirements will be used to produce the production image.
+However we now havehave ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file generated
+automatically and committd in the requirements folder based on the set of all latest working and tested
+requirement versions. Those requirement.txt files can be used as constraints file when installing apache
+airflow - either from the sources ``pip install -e . --constraint requirements/requirements-python3.7.txt``
+or from the pypi package ``pip install apache-airflow --constraint requirements/requirements-python3.7.txt``.
+This work also with extras - ``pip install .[gcp] --constraint requirements/requirements-python3.7.txt``.
+It is also possible to use constraints directly from github (note that it only works starting from
+airflow 1.10.10 version:
Review comment:
rephrased
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7841: Requirements
now depend on python version
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#issuecomment-603512068
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=h1) Report
> Merging [#7841](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/1982c3fdca1f04cfc41fc5b5e285d8f01c6b76ab&el=desc) will **decrease** coverage by `0.75%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7841 +/- ##
==========================================
- Coverage 86.97% 86.22% -0.76%
==========================================
Files 927 927
Lines 44963 44963
==========================================
- Hits 39108 38769 -339
- Misses 5855 6194 +339
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.25% <0.00%> (-72.50%)` | :arrow_down: |
| [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0.00%> (-64.16%)` | :arrow_down: |
| [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `39.28% <0.00%> (-60.72%)` | :arrow_down: |
| [airflow/api/auth/backend/kerberos\_auth.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9hcGkvYXV0aC9iYWNrZW5kL2tlcmJlcm9zX2F1dGgucHk=) | `28.16% <0.00%> (-54.93%)` | :arrow_down: |
| [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `47.82% <0.00%> (-52.18%)` | :arrow_down: |
| [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
| [airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==) | `53.33% <0.00%> (-46.67%)` | :arrow_down: |
| [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `30.43% <0.00%> (-45.66%)` | :arrow_down: |
| [airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5) | `55.00% <0.00%> (-45.00%)` | :arrow_down: |
| [airflow/providers/redis/sensors/redis\_key.py](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvc2Vuc29ycy9yZWRpc19rZXkucHk=) | `61.53% <0.00%> (-38.47%)` | :arrow_down: |
| ... and [12 more](https://codecov.io/gh/apache/airflow/pull/7841/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=footer). Last update [1982c3f...c1660bd](https://codecov.io/gh/apache/airflow/pull/7841?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397603139
##########
File path: CONTRIBUTING.rst
##########
@@ -341,31 +341,59 @@ be open to allow several different libraries with the same requirements to be in
The problem is that Apache Airflow is a bit of both - application to install and library to be used when
you are developing your own operators and DAGs.
-This - seemingly unsolvable - puzzle is solved as follows:
+This - seemingly unsolvable - puzzle is solved by having pinned requirement files. Those are available
+as of airflow 1.10.10.
Review comment:
Fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397603010
##########
File path: CONTRIBUTING.rst
##########
@@ -341,31 +341,59 @@ be open to allow several different libraries with the same requirements to be in
The problem is that Apache Airflow is a bit of both - application to install and library to be used when
you are developing your own operators and DAGs.
-This - seemingly unsolvable - puzzle is solved as follows:
+This - seemingly unsolvable - puzzle is solved by having pinned requirement files. Those are available
+as of airflow 1.10.10.
-* by default when you install ``apache-airflow`` package - the dependencies are as open as possible while
- still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
- install in case a direct or transitive dependency is released that breaks the installation. In such case
- when installing ``apache-airflow``, you might need to provide additional constraints (for
- example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
+Pinned requirement files
+------------------------
-* we have ``requirements.txt`` file generated automatically based on the set of all latest working
- and tested requirement versions. You can also use that file as a constraints file when installing
- apache airflow - either from the sources ``pip install -e . --constraint requirements.txt`` or
- from the pypi package ``pip install apache-airflow --constraint requirements.txt``. Note that
- this will also work with extras for example ``pip install .[gcp] --constraint requirements.txt`` or
- ``pip install apache-airflow[gcp] --constraint requirements.txt``
+By default when you install ``apache-airflow`` package - the dependencies are as open as possible while
+still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
+install in case a direct or transitive dependency is released that breaks the installation. In such case
+when installing ``apache-airflow``, you might need to provide additional constraints (for
+example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
-The ``requirements.txt`` file should be updated automatically via pre-commit whenever you update dependencies
-It reflects the current set of dependencies installed in the CI image of Apache Airflow.
-The same set of requirements will be used to produce the production image.
+However we now havehave ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file generated
+automatically and committd in the requirements folder based on the set of all latest working and tested
+requirement versions. Those requirement.txt files can be used as constraints file when installing apache
+airflow - either from the sources ``pip install -e . --constraint requirements/requirements-python3.7.txt``
+or from the pypi package ``pip install apache-airflow --constraint requirements/requirements-python3.7.txt``.
+This work also with extras - ``pip install .[gcp] --constraint requirements/requirements-python3.7.txt``.
+It is also possible to use constraints directly from github (note that it only works starting from
+airflow 1.10.10 version:
-If you do not use pre-commits and the CI builds fails / you need to regenerate it, you can do it manually:
-``pre-commit run generate-requirements --all-files`` or via script
-``./scripts/ci/ci_generate_requirements.sh``.
-This will try to regenerate the requirements.txt file with the latest requirements matching
-the setup.py constraints.
+.. code-block:: bash
+
+ pip install apache-airflow[gcp]==1.10.10 \
+ --constraint https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt
+
+There are different set of fixed requirements for different python major/minor versions and you should
+use the right python version file for the right python version.
+
+The ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file MUST be regenerated every time after setup.py
+is updated. This is checked automatically in Travis CI build there are separate jobs for each
+python version that checks if the requirements should be updated.
+
+If they are not you should regenerate the requirements locally using one of the methods described below.
Review comment:
Fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r396803764
##########
File path: .travis.yml
##########
@@ -41,24 +41,24 @@ jobs:
stage: pre-test
script: ./scripts/ci/ci_run_all_static_checks.sh
env: >-
- PYTHON_VERSION=3.6
+ PYTHON_SHORT_VERSION=3.6
Review comment:
Is there a reason we renamed PYTHON_VERSION to PYTHON_SHORT_VERSION?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7841:
Requirements now depend on python version
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7841: Requirements now depend on python version
URL: https://github.com/apache/airflow/pull/7841#discussion_r397602189
##########
File path: CONTRIBUTING.rst
##########
@@ -341,31 +341,59 @@ be open to allow several different libraries with the same requirements to be in
The problem is that Apache Airflow is a bit of both - application to install and library to be used when
you are developing your own operators and DAGs.
-This - seemingly unsolvable - puzzle is solved as follows:
+This - seemingly unsolvable - puzzle is solved by having pinned requirement files. Those are available
+as of airflow 1.10.10.
-* by default when you install ``apache-airflow`` package - the dependencies are as open as possible while
- still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
- install in case a direct or transitive dependency is released that breaks the installation. In such case
- when installing ``apache-airflow``, you might need to provide additional constraints (for
- example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
+Pinned requirement files
+------------------------
-* we have ``requirements.txt`` file generated automatically based on the set of all latest working
- and tested requirement versions. You can also use that file as a constraints file when installing
- apache airflow - either from the sources ``pip install -e . --constraint requirements.txt`` or
- from the pypi package ``pip install apache-airflow --constraint requirements.txt``. Note that
- this will also work with extras for example ``pip install .[gcp] --constraint requirements.txt`` or
- ``pip install apache-airflow[gcp] --constraint requirements.txt``
+By default when you install ``apache-airflow`` package - the dependencies are as open as possible while
+still allowing the apache-airflow to install. This means that 'apache-airflow' package might fail to
+install in case a direct or transitive dependency is released that breaks the installation. In such case
+when installing ``apache-airflow``, you might need to provide additional constraints (for
+example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``)
-The ``requirements.txt`` file should be updated automatically via pre-commit whenever you update dependencies
-It reflects the current set of dependencies installed in the CI image of Apache Airflow.
-The same set of requirements will be used to produce the production image.
+However we now havehave ``requirements-python<PYTHON_SHORT_VERSION>.txt`` file generated
+automatically and committd in the requirements folder based on the set of all latest working and tested
+requirement versions. Those requirement.txt files can be used as constraints file when installing apache
Review comment:
Fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services