You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2025/05/06 23:15:32 UTC
[PR] Remove limits for aiobotocore limiting boto3 [airflow]
potiuk opened a new pull request, #50285:
URL: https://github.com/apache/airflow/pull/50285
Those limits are causing pip to not install latest amazon provider when upgrading to latest versions in general.
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove limits for aiobotocore limiting boto3 [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #50285:
URL: https://github.com/apache/airflow/pull/50285#discussion_r2076551469
##########
providers/amazon/pyproject.toml:
##########
@@ -100,9 +100,6 @@ dependencies = [
# boto3 have native async support and we move away from aio aiobotocore
"aiobotocore" = [
"aiobotocore[boto3]>=2.21.1",
- # boto3 here should be synchronized with latest aiobotocore version otherwise pip might get
- # into a backtracking loop and fail to install the package
Review Comment:
We can still manually update constraints before 3.0.1 is out and force the amazon provider, but I **think** if we remove those (already outdated) limits - things should just work automatically.
The issue is really that:
a) this was really a hack to disable backtracking
b) 5 days ago aiobotocore released a new version (2.22.0) and it has
```
boto3>=1.37.2, <1.37.4
```
and we do not have it updated in those limits.
So when we are installing `aiobotocore` extra, 1.37.3 is GOOD version of boto3 -> but the limits in amazon provider 9.6.1 and 9.7.0rc1 do not allow it to be instaled together with 1.37.3 boto3.
By removing the limits from `aiobotocore` extra - we remove the outdated and really "artifficial` limit and allow to install aibotocore 2.22.0 and amazon provider 9.7.0rc2 (providing that we will release it without those limits)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove limits for aiobotocore limiting boto3 [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed pull request #50285: Remove limits for aiobotocore limiting boto3
URL: https://github.com/apache/airflow/pull/50285
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove limits for aiobotocore limiting boto3 [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #50285:
URL: https://github.com/apache/airflow/pull/50285#issuecomment-2856454600
I think I found out why in latest RC image of Airflow, we do not get latest amazon provider installed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove limits for aiobotocore limiting boto3 [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #50285:
URL: https://github.com/apache/airflow/pull/50285
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove limits for aiobotocore limiting boto3 [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #50285:
URL: https://github.com/apache/airflow/pull/50285#discussion_r2076527812
##########
providers/amazon/pyproject.toml:
##########
@@ -100,9 +100,6 @@ dependencies = [
# boto3 have native async support and we move away from aio aiobotocore
"aiobotocore" = [
"aiobotocore[boto3]>=2.21.1",
- # boto3 here should be synchronized with latest aiobotocore version otherwise pip might get
- # into a backtracking loop and fail to install the package
Review Comment:
So this is no longer a risK? Because we're using uv? I don't think I have enough context to fully review this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove limits for aiobotocore limiting boto3 [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #50285:
URL: https://github.com/apache/airflow/pull/50285#issuecomment-2856588272
Reopened to rebuild it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove limits for aiobotocore limiting boto3 [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #50285:
URL: https://github.com/apache/airflow/pull/50285#discussion_r2076545145
##########
providers/amazon/pyproject.toml:
##########
@@ -100,9 +100,6 @@ dependencies = [
# boto3 have native async support and we move away from aio aiobotocore
"aiobotocore" = [
"aiobotocore[boto3]>=2.21.1",
- # boto3 here should be synchronized with latest aiobotocore version otherwise pip might get
- # into a backtracking loop and fail to install the package
Review Comment:
No - it was just to solve `pip` backtracking - and we also use `pip 25.1` that **should** manage it better - we will see when the PR completes actually.
The problem with having those limits (I believe) is that (and this is what triggered my investigation) currently even amazon-provider 9.6.1 is not installed in the 3.0.1rc1 by default - it is `blocked ` at 9.6.0 it seems - and it's not because it cannot be installed (it can) but when we install `aiobotocore` and `amazon` together - pip apparently will have problem with finding the right
We will likely have to release an RC2 version of amazon provider @eladkal to hande that.
See yourself:
```
docker run -it apache/airflow:3.0.1rc1 providers list
package_name | description | version
==========================================+==============================================================================================+=========
apache-airflow-providers-amazon | Amazon integration (including Amazon Web Services (AWS) https://aws.amazon.com/) | 9.6.0
apache-airflow-providers-celery | Celery https://docs.celeryq.dev/en/stable/ | 3.10.6
apache-airflow-providers-cncf-kubernetes | Kubernetes https://kubernetes.io/ | 10.4.3
apache-airflow-providers-common-compat | Common Compatibility Provider - providing compatibility code for previous Airflow versions | 1.6.1
apache-airflow-providers-common-io | Common IO Provider | 1.5.4
apache-airflow-providers-common-messaging | Common Messaging Provider | 1.0.1rc1
apache-airflow-providers-common-sql | Common SQL Provider https://en.wikipedia.org/wiki/SQL | 1.26.0
apache-airflow-providers-docker | Docker https://www.docker.com/ | 4.3.1
apache-airflow-providers-elasticsearch | Elasticsearch https://www.elastic.co/elasticsearch | 6.2.2
apache-airflow-providers-fab | Flask App Builder https://flask-appbuilder.readthedocs.io/ | 2.0.2
apache-airflow-providers-ftp | File Transfer Protocol (FTP) https://tools.ietf.org/html/rfc114 | 3.12.3
apache-airflow-providers-git | Distributed version control system (GIT) https://git-scm.com/ | 0.0.2
apache-airflow-providers-google | Google services including: | 15.1.0
| |
| - Google Ads https://ads.google.com/ |
| - Google Cloud (GCP) https://cloud.google.com/ |
| - Google Firebase https://firebase.google.com/ |
| - Google LevelDB https://github.com/google/leveldb/ |
| - Google Marketing Platform https://marketingplatform.google.com/ |
| - Google Workspace https://workspace.google.com/ (formerly Google Suite) |
apache-airflow-providers-grpc | gRPC https://grpc.io/ | 3.7.3
apache-airflow-providers-hashicorp | Hashicorp including Hashicorp Vault https://www.vaultproject.io/ | 4.1.1
apache-airflow-providers-http | Hypertext Transfer Protocol (HTTP) https://www.w3.org/Protocols/ | 5.2.2
apache-airflow-providers-microsoft-azure | Microsoft Azure https://azure.microsoft.com/ | 12.3.1
apache-airflow-providers-mysql | MySQL https://www.mysql.com/ | 6.2.2
apache-airflow-providers-odbc | ODBC https://github.com/mkleehammer/pyodbc/wiki | 4.9.2
apache-airflow-providers-openlineage | OpenLineage https://openlineage.io/ | 2.2.0
apache-airflow-providers-postgres | PostgreSQL https://www.postgresql.org/ | 6.1.3
apache-airflow-providers-redis | Redis https://redis.io/ | 4.0.2
apache-airflow-providers-sendgrid | Sendgrid https://sendgrid.com/ | 4.0.1
apache-airflow-providers-sftp | SSH File Transfer Protocol (SFTP) https://tools.ietf.org/wg/secsh/draft-ietf-secsh-filexfer/ | 5.2.1
apache-airflow-providers-slack | Slack https://slack.com/ services integration including: | 9.0.5
| |
| - Slack API https://api.slack.com/ |
| - Slack Incoming Webhook https://api.slack.com/messaging/webhooks |
apache-airflow-providers-smtp | Simple Mail Transfer Protocol (SMTP) https://tools.ietf.org/html/rfc5321 | 2.0.3
apache-airflow-providers-snowflake | Snowflake https://www.snowflake.com/ | 6.2.2
apache-airflow-providers-ssh | Secure Shell (SSH) https://tools.ietf.org/html/rfc4251 | 4.0.1
apache-airflow-providers-standard | Airflow Standard Provider | 1.1.0
```
And when you are trying to force-install either 9.6.1 or 9.7.0rc1 you get this:
```
The conflict is caused by:
apache-airflow-providers-amazon 9.6.1 depends on boto3>=1.37.0
watchtower 3.4.0 depends on boto3<2 and >=1.9.253
apache-airflow-providers-amazon[aiobotocore] 9.6.1 depends on boto3>=1.37.0
apache-airflow-providers-amazon[aiobotocore] 9.6.1 depends on boto3<1.37.2 and >=1.37.0; extra == "aiobotocore"
The user requested (constraint) boto3==1.37.3
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove limits for aiobotocore limiting boto3 [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #50285:
URL: https://github.com/apache/airflow/pull/50285#issuecomment-2857869937
All checks passed :) .. so merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove limits for aiobotocore limiting boto3 [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #50285:
URL: https://github.com/apache/airflow/pull/50285#discussion_r2076574981
##########
providers/amazon/pyproject.toml:
##########
@@ -100,9 +100,6 @@ dependencies = [
# boto3 have native async support and we move away from aio aiobotocore
"aiobotocore" = [
"aiobotocore[boto3]>=2.21.1",
- # boto3 here should be synchronized with latest aiobotocore version otherwise pip might get
- # into a backtracking loop and fail to install the package
Review Comment:
Great, thanks for the context. Glad we can get ride of this one, I have hated that we've been carrying it so long.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org