You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2023/12/12 15:58:09 UTC
[I] Add unit tests to retrieve fsspec from providers including backwards compatibility [airflow]
potiuk opened a new issue, #36187:
URL: https://github.com/apache/airflow/issues/36187
### Body
We currently miss fsspec retrieval for providers and #36186 fixed compatibility issue with it, so we should likely add unit tests covering it.
### Committer
- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add unit tests to retrieve fsspec from providers including backwards compatibility [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #36187:
URL: https://github.com/apache/airflow/issues/36187#issuecomment-1852897234
> Well maybe not as they are existing providers, unlike common.io. Still it feels weird. If a change ends up in a provider before generally usable and still being worked on we should have a different mechanism IMHO.
Absolutely agree. Any idea how to do it? what do you think will be a better approach here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add unit tests to retrieve fsspec from providers including backwards compatibility [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #36187:
URL: https://github.com/apache/airflow/issues/36187#issuecomment-1852905056
> We should indeed test, but should we not yank the older released providers? This seems over doing the backwards compat (and maintaining it!) While the feature , somewhat arguably, couldn't be used.
This would not work.
Yanking is when there is an error, and we already have some of those providers as part of "golden" constraints in some versions of airflow, which means that when someone upgrades just airflow without providers, they will still remain with those versions. Yanking only prevents people from making specific version available for `pip` resolver when it resolves new set of packages to install, but when you either specified the version explicitly (which we do with constraints) or when the package is already installed `yanking` has no effect.
So in scenario when someone installed `airflow[amazon]==2.7.3` and after that `airflow==2.8.0` they would end up with broken amazon provider.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add unit tests to retrieve fsspec from providers including backwards compatibility [airflow]
Posted by "bolkedebruin (via GitHub)" <gi...@apache.org>.
bolkedebruin closed issue #36187: Add unit tests to retrieve fsspec from providers including backwards compatibility
URL: https://github.com/apache/airflow/issues/36187
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add unit tests to retrieve fsspec from providers including backwards compatibility [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #36187:
URL: https://github.com/apache/airflow/issues/36187#issuecomment-1852328549
cc: @bolkedebruin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add unit tests to retrieve fsspec from providers including backwards compatibility [airflow]
Posted by "bolkedebruin (via GitHub)" <gi...@apache.org>.
bolkedebruin commented on issue #36187:
URL: https://github.com/apache/airflow/issues/36187#issuecomment-1852883499
We should indeed test, but should we not yank the older released providers? This seems over doing the backwards compat (and maintaining it!) While the feature , somewhat arguably, couldn't be used.
Well maybe not as they are existing providers, unlike common.io. Still it feels weird. If a change ends up in a provider before generally usable and still being worked on we should have a different mechanism IMHO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org