You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/28 06:31:31 UTC
[GitHub] [airflow] RyanSiu1995 opened a new issue #13336: argcomplete is not compatible with importlib-metadata
RyanSiu1995 opened a new issue #13336:
URL: https://github.com/apache/airflow/issues/13336
**Apache Airflow version**: 1.10.14
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
**Environment**: Linux
- **Cloud provider or hardware configuration**:
- **OS** (e.g. from /etc/os-release):
- **Kernel** (e.g. `uname -a`):
- **Install tools**: pip-compile
- **Others**:
**What happened**:
I tried to compile the requirements with `pip-compile`. And I have found that `argcomplete==1.11.1` is not compatible with `importlib-metadata~=2.0`.
The logs are here.
```
Could not find a version that matches importlib-metadata<2,>=0.23,~=2.0 (from apache-airflow[github_enterprise,sentry]==1.10.14->-r requirements.in (line 4))
Tried: 0.0, 0.1, 0.1, 0.2, 0.2, 0.3, 0.3, 0.4, 0.4, 0.5, 0.5, 0.6, 0.6, 0.7, 0.7, 0.8, 0.8, 0.9, 0.9, 0.10, 0.10, 0.11, 0.11, 0.12, 0.12, 0.13, 0.13, 0.14, 0.14, 0.15, 0.15, 0.16, 0.16, 0.17, 0.17, 0.18, 0.18, 0.19, 0.19, 0.20, 0.20, 0.21, 0.21, 0.22, 0.22, 0.23, 0.23, 1.0.0, 1.0.0, 1.1.0, 1.1.0, 1.1.1, 1.1.1, 1.1.2, 1.1.2, 1.1.3, 1.1.3, 1.2.0, 1.2.0, 1.3.0, 1.3.0, 1.4.0, 1.4.0, 1.5.0, 1.5.0, 1.5.1, 1.5.1, 1.5.2, 1.5.2, 1.6.0, 1.6.0, 1.6.1, 1.6.1, 1.7.0, 1.7.0, 2.0.0, 2.0.0, 2.1.0, 2.1.0, 2.1.1, 2.1.1, 3.0.0, 3.0.0, 3.1.0, 3.1.0, 3.1.1, 3.1.1, 3.2.0, 3.2.0, 3.3.0, 3.3.0
There are incompatible versions in the resolved dependencies:
importlib-metadata~=2.0 (from apache-airflow[github_enterprise,sentry]==1.10.14->-r requirements.in (line 4))
importlib-metadata<2,>=0.23 (from argcomplete==1.11.1->apache-airflow[github_enterprise,sentry]==1.10.14->-r requirements.in (line 4))
```
**What you expected to happen**:
The dependency should not have conflict.
**How to reproduce it**:
pip-compile the airflow
**Anything else we need to know**:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #13336: argcomplete is not compatible with importlib-metadata
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #13336:
URL: https://github.com/apache/airflow/issues/13336#issuecomment-751638597
Unfortunately we do not support `pip-compile` installation. Airflow has fairly complex set of dependencies and the official way of installation is via PIP using constraint files:
http://airflow.apache.org/docs/apache-airflow/stable/installation.html#getting-airflow
For example for python 3.6 this is the right way of installing airflow (and it is stable to install):
```
pip install "apache-airflow==1.10.14" \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.6.txt"
```
In the constraint file for 1.10.14 you will find that we have (for python 3.6 - you can check other versions by changing python version in the URL):
* argcomplete==1.12.2
* importlib-metadata==2.1.0
We have a whole CI system implemented that verifies and automatically updates the constrainfts for upcoming Airflow relases and makes sure that those constraints are consistent (they pass `pip check`) and that all the 4000+ unit tests works with those constraints. Additionally those are constraints not requirements, which makes them stable to install, but at the same time they do not limit you from individually upgrading dependencies if needed (within the boundaries limited by Airflow|). We try to limit as little as possible because Airflow is both - application and library (there are people who need to upgrade their dependencies for the custom DAGs/Operators they write).
You can read more about Airflow's dependency management here and the reasoning why we are doing it: https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#dependency-management
Unfortunately neither `poetry` (we disscussed it recently) nor (as far as I know) `pip-compile` is flexible enough to give us what we want. They are opinionated in the way they treat dependencies and the models they introduce are not flexible enough to handle the case where you want to keep stable installation and leave the freedom to upgrade particular dependencies, that's why we have the `pip` + CI generated constraints solution which handle all the cases for us.
@ryansiu1995: If you know `pip-compile` well enough and want to continue using it and replicate what we have done with constraints, you are most welcome to describe and add alternative installation method also for others to use. However the per-python-version constratint files automatically generated in our repo should be the starting point for that, because we have quite sophisticated engineering behind to make it both consistent and automatically upgraded when new versions of packages are released.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #13336: argcomplete is not compatible with importlib-metadata
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #13336:
URL: https://github.com/apache/airflow/issues/13336
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #13336: argcomplete is not compatible with importlib-metadata
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #13336:
URL: https://github.com/apache/airflow/issues/13336#issuecomment-751602476
In Airflow 2.0 I think this problem has already been fixed as we run a pip check to detect all conflicts. Could you check it?
https://github.com/apache/airflow/blob/cb54513497f2738ae13c0cee8ad38db3d365f7e8/constraints-3.8.txt#L200
https://github.com/apache/airflow/blob/cb54513497f2738ae13c0cee8ad38db3d365f7e8/constraints-3.8.txt#L47
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #13336: argcomplete is not compatible with importlib-metadata
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #13336:
URL: https://github.com/apache/airflow/issues/13336#issuecomment-751638597
Unfortunately we do not support `pip-compile` installation. Airflow has fairly complex set of dependencies and the official way of installation is via PIP using constraint files:
http://airflow.apache.org/docs/apache-airflow/stable/installation.html#getting-airflow
For example for python 3.6 this is the right way of installing airflow (and it is stable to work):
```
pip install "apache-airflow==1.10.14" \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.6.txt"
```
In the constraint file for 1.10.14 you will find that we have (for python 3.6 - you can check other versions by changing python version in the URL):
* argcomplete==1.12.2
* importlib-metadata==2.1.0
We have a whole CI system implemented that verifies and automatically updates the constrainfts for upcoming Airflow relases and makes sure that those constraints are consistent (they pass `pip check`) and that all the 4000+ unit tests works with those constraints. Additionally those are constraints not requirements, which makes them stable to install, but at the same time they do not limit you from individually upgrading dependencies if needed (within the boundaries limited by Airflow. We try to limit as lilttle as possible because Airflow is both - application and library (there are people who need to upgrade their dependencies for the custom DAGs/Operators they write.
You can read more about Airflow's dependency management here and the reasoning why we are doing it: https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#dependency-management
Unfortunately neither `poetry` (we disscussed it recently) nor (as far as I know) `pip-compile` is flexible enough to give us what we want. They are opinionated in the way they treat dependencies and the models they introduce are not flexible enough to handle the case where you want to keep stable installation and leave the freedom to upgrade particular dependencies, that's why we have the `pip` + CI generated constraints solution which handle all the cases for us.
If you know `pip-compile` well enough and want to continue using it and replicate what we have done with constraints, you are most welcome to describe and add alternative installation method also for others to use. However the per-python-version constratint files automatically generated in our repo should be the starting point for that, because we have quite sophisticated engineering behind to make it both consistent and automatically upgraded when new versions of packages are released.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #13336: argcomplete is not compatible with importlib-metadata
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #13336:
URL: https://github.com/apache/airflow/issues/13336#issuecomment-751638597
Unfortunately we do not support `pip-compile` installation. Airflow has fairly complex set of dependencies and the official way of installation is via PIP using constraint files:
http://airflow.apache.org/docs/apache-airflow/stable/installation.html#getting-airflow
For example for python 3.6 this is the right way of installing airflow (and it is stable to work):
```
pip install "apache-airflow==1.10.14" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.6.txt"
```
In the constraint file for 1.10.14 you will find that we have (for python 3.6 - you can check other versions by changing python version in the URL):
* argcomplete==1.12.2
* importlib-metadata==2.1.0
We have a whole CI system implemented that verifies and automatically updates the constrainfts for upcoming Airflow relases and makes sure that those constraints are consistent (they pass `pip check`) and that all the 4000+ unit tests works with those constraints. Additionally those are constraints not requirements, which makes them stable to install, but at the same time they do not limit you from individually upgrading dependencies if needed (within the boundaries limited by Airflow. We try to limit as lilttle as possible because Airflow is both - application and library (there are people who need to upgrade their dependencies for the custom DAGs/Operators they write.
You can read more about Airflow's dependency management here and the reasoning why we are doing it: https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#dependency-management
Unfortunately neither `poetry` (we disscussed it recently) nor (as far as I know) `pip-compile` is flexible enough to give us what we want. They are opinionated in the way they treat dependencies and the models they introduce are not flexible enough to handle the case where you want to keep stable installation and leave the freedom to upgrade particular dependencies, that's why we have the 'pip' + CI generated constraints solution which handle all the cases for us.
If you know `pip-compile` well enough and want to continue using it and replicate what we have done with constraints, you are most welcome to describe and add alternative installation method also for others to use. However the per-python-version constratint files automatically generated in our repo should be the starting point for that, because we have quite sophisticated engineering behind to make it both consistent and automatically upgraded when new versions of packages are released.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #13336: argcomplete is not compatible with importlib-metadata
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #13336:
URL: https://github.com/apache/airflow/issues/13336#issuecomment-751638597
Unfortunately we do not support `pip-compile` installation. Airflow has fairly complex set of dependencies and the official way of installation is via PIP using constraint files:
http://airflow.apache.org/docs/apache-airflow/stable/installation.html#getting-airflow
For example for python 3.6 this is the right way of installing airflow (and it is stable to install):
```
pip install "apache-airflow==1.10.14" \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.6.txt"
```
In the constraint file for 1.10.14 you will find that we have (for python 3.6 - you can check other versions by changing python version in the URL):
* argcomplete==1.12.2
* importlib-metadata==2.1.0
We have a whole CI system implemented that verifies and automatically updates the constrainfts for upcoming Airflow relases and makes sure that those constraints are consistent (they pass `pip check`) and that all the 4000+ unit tests works with those constraints. Additionally those are constraints not requirements, which makes them stable to install, but at the same time they do not limit you from individually upgrading dependencies if needed (within the boundaries limited by Airflow|). We try to limit as little as possible because Airflow is both - application and library (there are people who need to upgrade their dependencies for the custom DAGs/Operators they write.
You can read more about Airflow's dependency management here and the reasoning why we are doing it: https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#dependency-management
Unfortunately neither `poetry` (we disscussed it recently) nor (as far as I know) `pip-compile` is flexible enough to give us what we want. They are opinionated in the way they treat dependencies and the models they introduce are not flexible enough to handle the case where you want to keep stable installation and leave the freedom to upgrade particular dependencies, that's why we have the `pip` + CI generated constraints solution which handle all the cases for us.
If you know `pip-compile` well enough and want to continue using it and replicate what we have done with constraints, you are most welcome to describe and add alternative installation method also for others to use. However the per-python-version constratint files automatically generated in our repo should be the starting point for that, because we have quite sophisticated engineering behind to make it both consistent and automatically upgraded when new versions of packages are released.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RyanSiu1995 commented on issue #13336: argcomplete is not compatible with importlib-metadata
Posted by GitBox <gi...@apache.org>.
RyanSiu1995 commented on issue #13336:
URL: https://github.com/apache/airflow/issues/13336#issuecomment-751602885
Unfortunately, I can only use 1.10.x right now.
So, I hope that this fix could be pulled back to 1.10.x
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #13336: argcomplete is not compatible with importlib-metadata
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #13336:
URL: https://github.com/apache/airflow/issues/13336#issuecomment-751638597
Unfortunately we do not support `pip-compile` installation. Airflow has fairly complex set of dependencies and the official way of installation is via PIP using constraint files:
http://airflow.apache.org/docs/apache-airflow/stable/installation.html#getting-airflow
For example for python 3.6 this is the right way of installing airflow (and it is stable to work):
```
pip install "apache-airflow==1.10.14" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.6.txt"
```
In the constraint file for 1.10.14 you will find that we have (for python 3.6 - you can check other versions by changing python version in the URL):
* argcomplete==1.12.2
* importlib-metadata==2.1.0
We have a whole CI system implemented that verifies and automatically updates the constrainfts for upcoming Airflow relases and makes sure that those constraints are consistent (they pass `pip check`) and that all the 4000+ unit tests works with those constraints. Additionally those are constraints not requirements, which makes them stable to install, but at the same time they do not limit you from individually upgrading dependencies if needed (within the boundaries limited by Airflow. We try to limit as lilttle as possible because Airflow is both - application and library (there are people who need to upgrade their dependencies for the custom DAGs/Operators they write.
You can read more about Airflow's dependency management here and the reasoning why we are doing it: https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#dependency-management
Unfortunately neither `poetry` (we disscussed it recently) nor (as far as I know) `pip-compile` is flexible enough to give us what we want. They are opinionated in the way they treat dependencies and the models they introduce are not flexible enough to handle the case where you want to keep stable installation and leave the freedom to upgrade particular dependencies, that's why we have the `pip` + CI generated constraints solution which handle all the cases for us.
If you know `pip-compile` well enough and want to continue using it and replicate what we have done with constraints, you are most welcome to describe and add alternative installation method also for others to use. However the per-python-version constratint files automatically generated in our repo should be the starting point for that, because we have quite sophisticated engineering behind to make it both consistent and automatically upgraded when new versions of packages are released.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #13336: argcomplete is not compatible with importlib-metadata
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #13336:
URL: https://github.com/apache/airflow/issues/13336#issuecomment-751639478
> In Airflow 2.0 I think this problem has already been fixed as we run a pip check to detect all conflicts. Could you check it?
@mik-laj -> FYI. we also run pip check in 1.10.14+ and the set of constraints there is also consistent.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #13336: argcomplete is not compatible with importlib-metadata
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #13336:
URL: https://github.com/apache/airflow/issues/13336#issuecomment-751638597
Unfortunately we do not support `pip-compile` installation. Airflow has fairly complex set of dependencies and the official way of installation is via PIP using constraint files:
http://airflow.apache.org/docs/apache-airflow/stable/installation.html#getting-airflow
For example for python 3.6 this is the right way of installing airflow (and it is stable to install):
```
pip install "apache-airflow==1.10.14" \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.6.txt"
```
In the constraint file for 1.10.14 you will find that we have (for python 3.6 - you can check other versions by changing python version in the URL):
* argcomplete==1.12.2
* importlib-metadata==2.1.0
We have a whole CI system implemented that verifies and automatically updates the constrainfts for upcoming Airflow relases and makes sure that those constraints are consistent (they pass `pip check`) and that all the 4000+ unit tests works with those constraints. Additionally those are constraints not requirements, which makes them stable to install, but at the same time they do not limit you from individually upgrading dependencies if needed (within the boundaries limited by Airflow. We try to limit as lilttle as possible because Airflow is both - application and library (there are people who need to upgrade their dependencies for the custom DAGs/Operators they write.
You can read more about Airflow's dependency management here and the reasoning why we are doing it: https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#dependency-management
Unfortunately neither `poetry` (we disscussed it recently) nor (as far as I know) `pip-compile` is flexible enough to give us what we want. They are opinionated in the way they treat dependencies and the models they introduce are not flexible enough to handle the case where you want to keep stable installation and leave the freedom to upgrade particular dependencies, that's why we have the `pip` + CI generated constraints solution which handle all the cases for us.
If you know `pip-compile` well enough and want to continue using it and replicate what we have done with constraints, you are most welcome to describe and add alternative installation method also for others to use. However the per-python-version constratint files automatically generated in our repo should be the starting point for that, because we have quite sophisticated engineering behind to make it both consistent and automatically upgraded when new versions of packages are released.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #13336: argcomplete is not compatible with importlib-metadata
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #13336:
URL: https://github.com/apache/airflow/issues/13336#issuecomment-751638597
Unfortunately we do not support `pip-compile` installation. Airflow has fairly complex set of dependencies and the official way of installation is via PIP using constraint files:
http://airflow.apache.org/docs/apache-airflow/stable/installation.html#getting-airflow
For example for python 3.6 this is the right way of installing airflow (and it is stable to install):
```
pip install "apache-airflow==1.10.14" \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.6.txt"
```
In the constraint file for 1.10.14 you will find that we have (for python 3.6 - you can check other versions by changing python version in the URL):
* argcomplete==1.12.2
* importlib-metadata==2.1.0
We have a whole CI system implemented that verifies and automatically updates the constrainfts for upcoming Airflow relases and makes sure that those constraints are consistent (they pass `pip check`) and that all the 4000+ unit tests works with those constraints. Additionally those are constraints not requirements, which makes them stable to install, but at the same time they do not limit you from individually upgrading dependencies if needed (within the boundaries limited by Airflow|). We try to limit as little as possible because Airflow is both - application and library (there are people who need to upgrade their dependencies for the custom DAGs/Operators they write).
You can read more about Airflow's dependency management here and the reasoning why we are doing it: https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#dependency-management
Unfortunately neither `poetry` (we disscussed it recently) nor (as far as I know) `pip-compile` is flexible enough to give us what we want. They are opinionated in the way they treat dependencies and the models they introduce are not flexible enough to handle the case where you want to keep stable installation and leave the freedom to upgrade particular dependencies, that's why we have the `pip` + CI generated constraints solution which handle all the cases for us.
If you know `pip-compile` well enough and want to continue using it and replicate what we have done with constraints, you are most welcome to describe and add alternative installation method also for others to use. However the per-python-version constratint files automatically generated in our repo should be the starting point for that, because we have quite sophisticated engineering behind to make it both consistent and automatically upgraded when new versions of packages are released.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #13336: argcomplete is not compatible with importlib-metadata
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #13336:
URL: https://github.com/apache/airflow/issues/13336#issuecomment-751639478
> In Airflow 2.0 I think this problem has already been fixed as we run a pip check to detect all conflicts. Could you check it?
@mik-laj -> FYI. we also run pip check in 1.10.14 and the set of constraints there is also consistent.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org