You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "notatallshaw (via GitHub)" <gi...@apache.org> on 2024/04/18 03:24:48 UTC
[I] Put reasonable lower bounds on providers in Airflow's extras [airflow]
notatallshaw opened a new issue, #39100:
URL: https://github.com/apache/airflow/issues/39100
### Description
Airflow should set lower bounds on the providers it specifies by extras.
### Use case/motivation
Airflow provides a number of extras that point to providers, e.g. `apache-beam` points to the provider `apache-airflow-providers-apache-beam` , but in the metadata it does this completely unbounded:
```
Provides-Extra: apache-beam
Requires-Dist: apache-airflow-providers-apache-beam; (python_version != "3.12") and extra == 'apache-beam'
```
This leads to the problem that dependency resolution tools might backtrack to `apache-airflow-providers-apache-beam==1.0.0` when trying to find solutions, which can cause resolution failures, e.g. https://github.com/astral-sh/uv/issues/3078
I don't think the lower bounds need to even be particularly strong, just avoid letting resolution tools attempt extremely old versions of providers.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Put reasonable lower bounds on providers in Airflow's extras [airflow]
Posted by "notatallshaw (via GitHub)" <gi...@apache.org>.
notatallshaw commented on issue #39100:
URL: https://github.com/apache/airflow/issues/39100#issuecomment-2062925697
cc @potiuk digging deep into resolutions (https://github.com/astral-sh/uv/issues/3078) this seems like it would be sinigifcantly easier recommend uv to end users as a way of installing airflow, at least in the current way uv can end up aggressively checking older versions of (transitive) dependencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Put reasonable lower bounds on providers in Airflow's extras [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #39100:
URL: https://github.com/apache/airflow/issues/39100#issuecomment-4149614216
Closing as to my understanding we don't have an open task here and by now we have a good mechanism of how to set lower bounds for libraries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Put reasonable lower bounds on providers in Airflow's extras [airflow]
Posted by "notatallshaw (via GitHub)" <gi...@apache.org>.
notatallshaw commented on issue #39100:
URL: https://github.com/apache/airflow/issues/39100#issuecomment-2064146273
I am happy to put a well thought out proposal together on the mailing list, but beyond that I don't think I can either lead or implement it.
I'm pretty much at capacity with my OSS work, I already have a patch for Airflow I [promised](https://github.com/apache/airflow/discussions/33052#discussioncomment-6813454) to turn into a PR last year and I've not managed to get round to doing (in part delayed because of company bureaucracy, in part I started spending a lot more focus on Python's packaging standards and it ate up all my OSS energy).
But if putting a proposal on the mailing list is a good start, without commiting to implementing it myself, I will do that soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Put reasonable lower bounds on providers in Airflow's extras [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #39100:
URL: https://github.com/apache/airflow/issues/39100#issuecomment-2064053468
Yes I think it might make sense to put some limits, but it's hard to say what those limits should be.
Generally speaking, the point of having old providers (even very old) and new Airflow is the whole point of separating providers and Airflow - simply, if provider has breaking changes, users can still upgrade Airflow but keep **some** providers not updated to latest versions.
The question is: "DEFINE REASONABLE". We already have some rules that we follow on providers supporting minimum Airflow versions - and those rules are well defined. https://github.com/apache/airflow/blob/main/PROVIDERS.rst#upgrading-minimum-supported-version-of-airflow
If someone will come up with a proposal for what would be "REASONABLE", I think it makes sense - the extra `limits` are not hard - they are soft, so it does not even "block" the users if they want to use older versions.
If so - maybe a good idea will be to raise the proposal as discusison in our devlist - this is where we discuss such ideas - for example the 12 months policy for providers have been discussed in:
* https://lists.apache.org/thread/csczm7xmnntdz9wtjbod8pqgt13zoggo
and lazy consensus proposed and achieved here:
* https://lists.apache.org/thread/43v1dnww37t0o924mzo365ot2p58k864
Similarly - discussion should be started, with reasoning and proposal, and then after achieving a consensus, we should implement it. I am all for it and I am also super-happy to guide someone doing it. IT's been a long time I was **doing** all these things myself, but this time I started to think about others stepping in and driving this - I am happy to support and guide implementation of it so if you feel like it @notatallshaw -> I would be super-happy to take the back seat on that one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Put reasonable lower bounds on providers in Airflow's extras [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal closed issue #39100: Put reasonable lower bounds on providers in Airflow's extras
URL: https://github.com/apache/airflow/issues/39100
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Put reasonable lower bounds on providers in Airflow's extras [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #39100:
URL: https://github.com/apache/airflow/issues/39100#issuecomment-2062923137
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Put reasonable lower bounds on providers in Airflow's extras [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #39100:
URL: https://github.com/apache/airflow/issues/39100#issuecomment-2064164449
Sure. Let's start with that. Implementation might be simple and possibly someone else might volunteer - we already have a few people who know their ways around that code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org