You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/11/05 23:32:11 UTC
[GitHub] [airflow] mik-laj opened a new issue #12120: Very old libraries in constraints.txt
mik-laj opened a new issue #12120:
URL: https://github.com/apache/airflow/issues/12120
Dear and Wonderful Citizens,
I started to look at what libraries we have defined in the `constraints-*.txt` file and I am a bit surprised because we have this constraints defined on very old libraries.
https://github.com/apache/airflow/blob/053afe7/constraints-3.8.txt
Sometimes we have defined libraries that are over 3 years old, which can cause security problems. Old versions of the library may have vulnerabilities that have probably been fixed in newer versions.
I am most concerned about dependency conflicts. Old libraries are only compatible with old libraries, which can cause problems if the user wants to use a new version of the same library.
At the moment (5 November 2020), this is the status of our outdated packages: and their ages.
| package_name | current_version | latest_version | diff_part | age |
|------------------------------------|-------------------|------------------|-------------|------------------|
| clickclick | 1.2.2 | 20.10.2 | 1-major | 3 years |
| Markdown | 2.6.11 | 3.3.3 | 1-major | 2 years |
| dnspython | 1.16.0 | 2.0.0 | 1-major | 1 year, 7 months |
| mysqlclient | 1.3.14 | 2.0.1 | 1-major | 1 year, 6 months |
| vine | 1.3.0 | 5.0.0 | 1-major | 1 year, 5 months |
| isort | 4.3.21 | 5.6.4 | 1-major | 1 year, 3 months |
| google-cloud-language | 1.3.0 | 2.0.0 | 1-major | 1 year, 2 months |
| oauthlib | 2.1.0 | 3.1.0 | 1-major | 1 year, 2 months |
| watchtower | 0.7.3 | 1.0.0 | 1-major | 1 year, 2 months |
| docker | 3.7.3 | 4.3.1 | 1-major | 1 year, 2 months |
| azure-mgmt-containerinstance | 1.5.0 | 2.0.0 | 1-major | 1 year, 1 month |
| azure-storage-blob | 2.1.0 | 12.5.0 | 1-major | 1 year, 1 month |
| traitlets | 4.3.3 | 5.0.5 | 1-major | 1 year, 12 days |
| google-cloud-translate | 1.7.0 | 3.0.1 | 1-major | 10 months |
| google-cloud-speech | 1.3.2 | 2.0.0 | 1-major | 7 months |
| kubernetes | 11.0.0 | 12.0.0 | 1-major | 7 months |
| google-cloud-vision | 1.0.0 | 2.0.0 | 1-major | 7 months |
| Flask-Babel | 1.0.0 | 2.0.0 | 1-major | 6 months |
| freezegun | 0.3.15 | 1.0.0 | 1-major | 6 months |
| google-cloud-tasks | 1.5.0 | 2.0.0 | 1-major | 6 months |
| google-cloud-texttospeech | 1.0.1 | 2.2.0 | 1-major | 5 months |
| azure-kusto-data | 0.0.45 | 1.0.3 | 1-major | 5 months |
| google-cloud-kms | 1.4.0 | 2.2.0 | 1-major | 5 months |
| multidict | 4.7.6 | 5.0.0 | 1-major | 4 months |
| google-crc32c | 0.1.0 | 1.0.0 | 1-major | 4 months |
| google-cloud-datacatalog | 0.7.0 | 2.0.0 | 1-major | 4 months |
| google-cloud-automl | 1.0.1 | 2.1.0 | 1-major | 4 months |
| google-cloud-monitoring | 1.0.0 | 2.0.0 | 1-major | 4 months |
| google-cloud-redis | 1.0.0 | 2.0.0 | 1-major | 4 months |
| google-cloud-secret-manager | 1.0.0 | 2.0.0 | 1-major | 3 months |
| apispec | 3.3.1 | 4.0.0 | 1-major | 3 months |
| google-cloud-bigquery | 1.26.1 | 2.3.1 | 1-major | 3 months |
| celery | 4.4.7 | 5.0.2 | 1-major | 3 months |
| azure-cosmos | 3.2.0 | 4.2.0 | 1-major | 3 months |
| fastavro | 0.24.0 | 1.1.0 | 1-major | 3 months |
| google-cloud-container | 1.0.1 | 2.1.0 | 1-major | 2 months |
| google-cloud-bigquery-datatransfer | 1.1.0 | 2.1.0 | 1-major | 2 months |
| importlib-metadata | 1.7.0 | 2.0.0 | 1-major | 2 months |
| pyarrow | 1.0.0 | 2.0.0 | 1-major | 2 months |
| sphinxcontrib-spelling | 5.2.1 | 7.1.0 | 1-major | 2 months |
| google-cloud-dlp | 1.0.0 | 2.0.0 | 1-major | 2 months |
| kombu | 4.6.11 | 5.0.2 | 1-major | 2 months |
| google-cloud-pubsub | 1.7.0 | 2.1.0 | 1-major | 2 months |
| humanize | 2.6.0 | 3.1.0 | 1-major | 2 months |
| google-resumable-media | 0.7.1 | 1.1.0 | 1-major | a month |
| google-ads | 6.0.0 | 7.0.0 | 1-major | a month |
| azure-mgmt-resource | 10.2.0 | 15.0.0 | 1-major | a month |
| google-cloud-dataproc | 1.1.1 | 2.0.2 | 1-major | a month |
| vertica-python | 0.11.0 | 1.0.0 | 1-major | a month |
| amqp | 2.6.1 | 5.0.1 | 1-major | a month |
| pytest-xdist | 1.34.0 | 2.1.0 | 1-major | 28 days |
| portalocker | 1.7.1 | 2.0.0 | 1-major | 16 days |
| gunicorn | 19.10.0 | 20.0.4 | 1-major | 3 days |
I generated this table with the script:
https://gist.github.com/mik-laj/880b07bfbdbd5c65b4b2260f6c0fee72
CC: @potiuk
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12120: Old libraries in setup.py causing dependency resolution to pull old transitive constraints (3 years+)
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12120:
URL: https://github.com/apache/airflow/issues/12120#issuecomment-749110849
Also @HaloKo4 -> those are not 'requirements' - those are 'constraint' files. This means that when you are installing airflow you can use those constraint files and they will "guarantee" that the installation will produce non-conflicting set of dependencies (as of 1.10.14 and 2.0.0). We have a process in place where during our CI builds we automatically upgrade to newer constraints where possible (and verify if airflow works and tests pass). Those constraints work in the 'worst' case when you install all "ci" extras "devel_ci" - which basically means "everything".
You are still free to upgrade to newer version of any dependency for example if you only installed several extras and the constraints are only applied at airflow installation time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12120: Old libraries in setup.py causing dependency resolution to pull old transitive constraints (3 years+)
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12120:
URL: https://github.com/apache/airflow/issues/12120#issuecomment-1046107496
So it's more than 'month passed' :) . We actually implemented actions to address it :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pvcnt commented on issue #12120: Old libraries in setup.py causing dependency resolution to pull old transitive constraints (3 years+)
Posted by GitBox <gi...@apache.org>.
pvcnt commented on issue #12120:
URL: https://github.com/apache/airflow/issues/12120#issuecomment-734226095
There is even a conflict in Airflow 1.10.13 with importlib-metadata, because Airflow requires `importlib-metadata~=2.0`, while argcomplete (required at 1.12.0 by Airflow) requires `importlib-metadata<2,>=0.23`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] HaloKo4 commented on issue #12120: Old libraries in setup.py causing dependency resolution to pull old transitive constraints (3 years+)
Posted by GitBox <gi...@apache.org>.
HaloKo4 commented on issue #12120:
URL: https://github.com/apache/airflow/issues/12120#issuecomment-748585165
Where airflow even use dnspython ?
I see it as dependency for mongo provider but I don't see where this package is actually being used in the code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12120: Old libraries in setup.py causing dependency resolution to pull old transitive constraints (3 years+)
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12120:
URL: https://github.com/apache/airflow/issues/12120#issuecomment-932971058
I am not sure we should keep it as "open issue" if there are no clear tasks to complete.
I think the scope of the issue is very generic and it has no chance to ever end if it is defined the way it is defined. I think we should not have (or have very little) of generic issues such us "continuous update of something" without having a list of tasks this leads to.
The current approach for dependency management is well estabilished when it comes to automated upgrades::
* we try put no upper boundaries of dependencies
* unless we have to for some reason and we document what the reason is in setup.py
* the "constraint upgrade" mechanism is specifically designed to update the constrainst as the new versions are released
The limitations we have are in:
* setup.cfg when they are imposed by airlfow
* setup.py when they are imposed by one of the providers
* Dockerfile.ci/Dockerfile if they are imposed by imperfect `pip eager-upgrade` mechanism that will prevent the combination of `airflow + providers` from working correctly when the eager-update mechanism is invoked.
However we have no process currently (and this issue is not helping with it on how to get rid of the upper-bound limitations over time - this is done on an ad-hoc base.
I think most of those limitations are somewhaat documented (with URLs and explanation where they came from in the setup.py/.cfg/Dockerfiles (but possibly not all of them are). I believe we should have an explanation for every single upper-bound limit that we have. Also some of the reasons for those might be alreaady gone, but we do not know it, some of them might need some work to remove (for example recent celery 5 upgrade).
I think a good idea how to treat that one would be that someone ( might be a even a good first issue) reviews all the setup.py/setup.cfg/Dockerfile upper-bound limitations and verifies if the reasons are:
a) documented
b) still valid
c) maybe even we should create a separate issue type for such limitations
Then what we possibly need is some kind of periodic review of those "upper bound" - if we have list of issues for those, that might be actually possible to manage and contro and even periodically review them (I think we could even automate big parts of this when the new GitHub Issues are enabled for public projects).
So I think we should close that issue and create a bunch of issues (grouped together) for every single limitation we have now and figure out what we do with those - but it requires a non-trivial amount of checking/verifying the current limits (maybe some investigations as well on where they came from), documentting and possibly organising a bit the process of regular review.
Anyone :) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #12120: Old libraries in setup.py causing dependency resolution to pull old transitive constraints (3 years+)
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #12120:
URL: https://github.com/apache/airflow/issues/12120#issuecomment-723076730
I will change the title of the issue, because it is not the matter of constraints, but it's the matter of limitations in setup.py and transitional dependencies that we have for all those libraries.
There is nothing we can do on the constraint level. The constraints files are generated automatically by PIP dependency mechanisms - whatever PIP resolves using setup.py limits is automatically updated as new version of constraints.
So if we want to do anything about it (do we?) we should remove some of the limitations there and upgrade all the different providers/core dependencies we have to use the latest version of dependent libraries - basically for each provider separately. Even that will not help in some cases because the newest version of those libraries might transitively use some older versions of dependent libraries.
Does anyone have some proposal there? Should we somehow make an effort to upgrade those? Maybe there is someone who would like o lead that?
Note, that in order to do it, we likely need to have some system tests in place and implemented for those providers that we decide to bump to later version of dependent libraries (https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-4+Support+for+Automation+of+System+Tests+for+external+systems) because what we basically need to do is to upgrade the libraries that we already know usually that they have some compatibility issues (for example all the google providers will have to be sooner or later migrated to >2.0.0 python libraries and automated unit testing is not enough for those kinds of changes (those libraries are not backwards compatible).
WDYT others? Is it worth to make such a concerted effort? Are there some real benefits from that? Should we do it? For 2.0 or later?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek edited a comment on issue #12120: Old libraries in setup.py causing dependency resolution to pull old transitive constraints (3 years+)
Posted by GitBox <gi...@apache.org>.
turbaszek edited a comment on issue #12120:
URL: https://github.com/apache/airflow/issues/12120#issuecomment-734229419
Related #12508 #12636 #12635
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #12120: Old libraries in setup.py causing dependency resolution to pull old transitive constraints (3 years+)
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #12120:
URL: https://github.com/apache/airflow/issues/12120#issuecomment-931571206
@mik-laj do we have further tasks on this issue?
I'm not able to run the notebook you shared to get updated list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #12120: Old libraries in setup.py causing dependency resolution to pull old transitive constraints (3 years+)
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #12120:
URL: https://github.com/apache/airflow/issues/12120#issuecomment-748643688
@HaloKo4 We generate one constraints file to check compatibility with all libraries and detect all conflicts, so even if mongo only requires an older version, most users will have the old version of this library.
Here is the official Airflow installation guide, which installs libraries always the latest tested versions.
http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/installation.html
If you try to install Airflow and dnspython according to the instructions, the latest version of the dnspython library will not be installed.
```shell script
AIRFLOW_VERSION=master
PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
pip install \
--constraint "${CONSTRAINT_URL}" \
".[postgres,google]" \
dnspython
```
I found a ticket in the eventlet project and I hope that it will be solved.
https://github.com/eventlet/eventlet/issues/619
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #12120: Old libraries in setup.py causing dependency resolution to pull old transitive constraints (3 years+)
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #12120:
URL: https://github.com/apache/airflow/issues/12120#issuecomment-723084392
I agree with Jarek, that some of the constraints come from dependencies of Airflow's dependencies. And since we have lots of dependencies it is not always straightforward to support all versions as these dependencies define there own requirements of versions.
So while having latest and greatest would be awesome, we need to make a concise effort that we don't make Airflow incompatible with other dependencies since PIP resolver is not going to allow it any longer
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12120: Old libraries in setup.py causing dependency resolution to pull old transitive constraints (3 years+)
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12120:
URL: https://github.com/apache/airflow/issues/12120#issuecomment-723079843
BTW. If somebody want to see where the deps come from, it's easy to use pipdeptree:
[pipdeptree.3.8.txt](https://github.com/apache/airflow/files/5501434/pipdeptree.3.8.txt)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12120: Old libraries in setup.py causing dependency resolution to pull old transitive constraints (3 years+)
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12120:
URL: https://github.com/apache/airflow/issues/12120#issuecomment-1046107075
Actually - we already agreed, merged and achieved lazy consensus to try the approach that addresses that issue:
https://github.com/apache/airflow/pull/21356
We removed upper limits from most of the libraries and only really left those that we know why we are limiting them and what is the condition to remove those, which allows us to periodicallly (at minor relase time) to review whether the reasons for limiting are gone .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal closed issue #12120: Old libraries in setup.py causing dependency resolution to pull old transitive constraints (3 years+)
Posted by GitBox <gi...@apache.org>.
eladkal closed issue #12120:
URL: https://github.com/apache/airflow/issues/12120
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12120: Very old libraries in constraints.txt (3 years+)
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12120:
URL: https://github.com/apache/airflow/issues/12120#issuecomment-723076730
I will change the title of the issue, because it is not the matter of constraints, but it's the matter of limitations in setup.py and transitional dependencies that we have for all those libraries.
There is nothing we can do on the constraint level. The constraints files are generated automatically by PIP dependency mechanisms - whatever PIP resolves using setup.py limits is automatically updated as new version of constraints.
So if we want to do anything about it (do we?) we should remove some of the limitations there and upgrade all the different providers we have to use the latest version of dependent libraries - basically for each provider separately. Even that will not help in some cases because the newest version of those libraries might transitively use some older versions of dependent libraries.
Does anyone have some proposal there? Should we somehow make an effort to upgrade those? Maybe there is someone who would like o lead that?
Note, that in order to do it, we likely need to have some system tests in place and implemented for those providers that we decide to bump to later version of dependent libraries (https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-4+Support+for+Automation+of+System+Tests+for+external+systems) because what we basically need to do is to upgrade the libraries that we already know usually that they have some compatibility issues (for example all the google providers will have to be sooner or later migrated to >2.0.0 python libraries and automated unit testing is not enough for those kinds of changes (those libraries are not backwards compatible).
WDYT others? Is it worth to make such a concerted effort? Are there some real benefits from that? Should we do it? For 2.0 or later?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12120: Old libraries in setup.py causing dependency resolution to pull old transitive constraints (3 years+)
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12120:
URL: https://github.com/apache/airflow/issues/12120#issuecomment-734258886
I don't think we'd want to fix deps for 1.10.* Our focus in #12636 is to make them fixed (and non-breakable in the future for Airflow 2.0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #12120: Old libraries in setup.py causing dependency resolution to pull old transitive constraints (3 years+)
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #12120:
URL: https://github.com/apache/airflow/issues/12120#issuecomment-748643688
@HaloKo4 We generate one constraints file to check compatibility with all libraries and detect all conflicts, so even if mongo only requires an older version, most users will have the old version of this library.
Here is the official Airflow installation guide, which installs libraries always the latest tested versions.
http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/installation.html
If you try to install Airflow and dnspython according to the instructions, the latest version of the dnspython library will not be installed.
````
AIRFLOW_VERSION=master
PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
pip install \
--constraint "${CONSTRAINT_URL}" \
"apache-airflow[postgres,google]==${AIRFLOW_VERSION}" \
dnspython
```
I found a ticket in the eventlet project and I hope that it will be solved.
https://github.com/eventlet/eventlet/issues/619
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] HaloKo4 commented on issue #12120: Old libraries in setup.py causing dependency resolution to pull old transitive constraints (3 years+)
Posted by GitBox <gi...@apache.org>.
HaloKo4 commented on issue #12120:
URL: https://github.com/apache/airflow/issues/12120#issuecomment-748633885
but this dependency exist only on mongo provider. what am I missing here?
https://github.com/apache/airflow/blob/master/setup.py#L321
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #12120: Old libraries in setup.py causing dependency resolution to pull old transitive constraints (3 years+)
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #12120:
URL: https://github.com/apache/airflow/issues/12120#issuecomment-734258886
I don't think we'd want to fix deps for 1.10.* Our focus in #12636 is to make them fixed (and non-breakable in the future) for Airflow 2.0.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #12120: Old libraries in setup.py causing dependency resolution to pull old transitive constraints (3 years+)
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #12120:
URL: https://github.com/apache/airflow/issues/12120#issuecomment-734229419
Working on that #12508 #12636 #12635
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #12120: Old libraries in setup.py causing dependency resolution to pull old transitive constraints (3 years+)
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #12120:
URL: https://github.com/apache/airflow/issues/12120#issuecomment-748643688
@HaloKo4 We generate one constraints file to check compatibility with all libraries and detect all conflicts, so even if mongo only requires an older version, most users will have the old version of this library.
Here is the official Airflow installation guide, which installs libraries always the latest tested versions.
http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/installation.html
If you try to install Airflow and dnspython according to the instructions, the latest version of the dnspython library will not be installed.
```shell script
AIRFLOW_VERSION=master
PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
pip install \
--constraint "${CONSTRAINT_URL}" \
"apache-airflow[postgres,google]==${AIRFLOW_VERSION}" \
dnspython
```
I found a ticket in the eventlet project and I hope that it will be solved.
https://github.com/eventlet/eventlet/issues/619
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #12120: Old libraries in setup.py causing dependency resolution to pull old transitive constraints (3 years+)
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #12120:
URL: https://github.com/apache/airflow/issues/12120#issuecomment-748632079
Airflow doesn't use this dependency directly, but other dependencies that require Airflow do require this library. Python doesn't have dependency isolation (for example, [the node has](https://npm.github.io/how-npm-works-docs/npm2/how-npm2-works.html)), so we need to make sure other libraries don't restrict their dependency versions too much, otherwise users won't be able to use their favorite libraries in the latest versions. Users will also not be able to get security patches if the version restrictions are too restrictive.
```
pipdeptree -r -p dnspython
dnspython==1.16.0
- email-validator==1.1.2 [requires: dnspython>=1.15.0]
- Flask-AppBuilder==3.1.1 [requires: email-validator>=1.0.5,<2]
- apache-airflow==2.0.0 [requires: flask-appbuilder~=3.1.1]
- eventlet==0.29.1 [requires: dnspython>=1.15.0,<2.0.0]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #12120: Old libraries in setup.py causing dependency resolution to pull old transitive constraints (3 years+)
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #12120:
URL: https://github.com/apache/airflow/issues/12120#issuecomment-748632079
Airflow doesn't use this dependency directly, but other dependencies that require Airflow do require this library. Python doesn't have dependency isolation (for example, [the node has](https://npm.github.io/how-npm-works-docs/npm2/how-npm2-works.html)), so we need to make sure other libraries don't restrict their dependency versions too much, otherwise users won't be able to use their favorite libraries in the latest versions. Users will also not be able to get security patches if the version restrictions are too restrictive.
```
pipdeptree -r -p dnspython
dnspython==1.16.0
- email-validator==1.1.2 [requires: dnspython>=1.15.0]
- Flask-AppBuilder==3.1.1 [requires: email-validator>=1.0.5,<2]
- apache-airflow==2.0.0 [requires: flask-appbuilder~=3.1.1]
- eventlet==0.29.1 [requires: dnspython>=1.15.0,<2.0.0]
```
In this case, it looks like we need to check why eventlet requires a version older than 2.0.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #12120: Old libraries in setup.py causing dependency resolution to pull old transitive constraints (3 years+)
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #12120:
URL: https://github.com/apache/airflow/issues/12120#issuecomment-932971058
I am not sure we should keep it as "open issue" if there are no clear tasks to complete.
I think the scope of the issue is very generic and it has no chance to ever end if it is defined the way it is defined. I think we should not have (or have very little) of generic issues such us "continuous update of something" without having a list of tasks this leads to.
The current approach for dependency management is well estabilished when it comes to automated upgrades::
* we try put no upper boundaries of dependencies
* unless we have to for some reason and we document what the reason is in setup.py
* the "constraint upgrade" mechanism is specifically designed to update the constrainst as the new versions are released
The limitations we have are in:
* setup.cfg when they are imposed by airlfow
* setup.py when they are imposed by one of the providers
* Dockerfile.ci/Dockerfile if they are imposed by imperfect `pip eager-upgrade` mechanism that will prevent the combination of `airflow + providers` from working correctly when the eager-update mechanism is invoked.
However we have no process currently (and this issue is not helping with it) on how to get rid of the upper-bound limitations over time - this is done on an ad-hoc base.
I think most of those limitations are somewhaat documented (with URLs and explanation where they came from in the setup.py/.cfg/Dockerfiles (but possibly not all of them are). I believe we should have an explanation for every single upper-bound limit that we have. Also some of the reasons for those might be alreaady gone, but we do not know it, some of them might need some work to remove (for example recent celery 5 upgrade).
I think a good idea how to treat that one would be that someone ( might be a even a good first issue) reviews all the setup.py/setup.cfg/Dockerfile upper-bound limitations and verifies if the reasons are:
a) documented
b) still valid
c) maybe even we should create a separate issue type for such limitations
Then what we possibly need is some kind of periodic review of those "upper bound" - if we have list of issues for those, that might be actually possible to manage and contro and even periodically review them (I think we could even automate big parts of this when the new GitHub Issues are enabled for public projects).
So I think we should close that issue and create a bunch of issues (grouped together) for every single limitation we have now and figure out what we do with those - but it requires a non-trivial amount of checking/verifying the current limits (maybe some investigations as well on where they came from), documentting and possibly organising a bit the process of regular review.
Anyone :) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #12120: Old libraries in setup.py causing dependency resolution to pull old transitive constraints (3 years+)
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #12120:
URL: https://github.com/apache/airflow/issues/12120#issuecomment-1046102122
I agree with you Jarek and since the months passed without further comments I'm closing the issue.
If needed lets open new issues with defined scope
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org