You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/06/14 21:38:18 UTC
[GitHub] [airflow] potiuk opened a new pull request, #24454: Update description of installing providers separately from core
potiuk opened a new pull request, #24454:
URL: https://github.com/apache/airflow/pull/24454
The description of how to install providers separately from the
core was wrong. It suggested installing main constraints, but
in fact we are not able to give any guarantees when installing
providers this way so no constraints can guarantee consistent
set of dependencies and the user is on their own in this case.
This PR corrects the installation docs.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragement file, named `{pr_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on pull request #24454: Update description of installing providers separately from core
Posted by GitBox <gi...@apache.org>.
eladkal commented on PR #24454:
URL: https://github.com/apache/airflow/pull/24454#issuecomment-1155743859
What is the effort in creating:
Constraint-2.3.2-latest?
Currently we have constraint-2.3.2 which created one time only when version is released (so providers versions are never updated). Using constraint-main may cause problems as you deacribed in this PR.
Basicly my idea/question/suggestion is to have a latest version of updated constraints per specific core version which will be updated regularly. This will save users the big headeach if doable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #24454: Update description of installing providers separately from core
Posted by GitBox <gi...@apache.org>.
potiuk merged PR #24454:
URL: https://github.com/apache/airflow/pull/24454
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #24454: Update description of installing providers separately from core
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #24454:
URL: https://github.com/apache/airflow/pull/24454#issuecomment-1160034255
The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #24454: Update description of installing providers separately from core
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24454:
URL: https://github.com/apache/airflow/pull/24454#issuecomment-1155737039
This has been result of talking to a user :).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a diff in pull request #24454: Update description of installing providers separately from core
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on code in PR #24454:
URL: https://github.com/apache/airflow/pull/24454#discussion_r897349353
##########
docs/apache-airflow/installation/installing-from-pypi.rst:
##########
@@ -127,33 +127,27 @@ being installed.
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
pip install "apache-airflow[postgres,google]==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"
-Installation and upgrading of Airflow providers separately
-==========================================================
+Installing/upgrading/downgrading providers separately from Airflow core
+=======================================================================
-You can manually install all the providers you need. You can continue using the "providers" constraint files
-but the 'versioned' airflow constraints installs only the versions of providers that were available in PyPI at
-the time of preparing of the airflow version. However, usually you can use "main" version of the providers
-to install latest version of providers. Usually the providers work with most versions of Airflow, if there
-will be any incompatibilities, it will be captured as package dependencies.
+In order to add new features, implement bug-fixes or simply maintain backwards compatibility, you might need
+to install, upgrade or downgrade any of the providers you need. We release providers independently from the
+core of Airflow, so often new versions of providers are released before Airflow is, also if you do not want
+yet to upgrade Airflow to the latest version, you might want to install newly released providers separately.
Review Comment:
```suggestion
to install, upgrade or downgrade any of the providers you need. We release providers independently from
core Airflow, so often new versions of providers are released before Airflow is. You also may might want to
install newly released providers separately without also upgrading your core Airflow version.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #24454: Update description of installing providers separately from core
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24454:
URL: https://github.com/apache/airflow/pull/24454#issuecomment-1159811785
Should we merge it @eladkal ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #24454: Update description of installing providers separately from core
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24454:
URL: https://github.com/apache/airflow/pull/24454#issuecomment-1155774282
> What is the effort in creating:
> Constraint-2.3.2-latest?
1st of all this is not for installing latest providers on "latest" airflow, but installing them on ANY compatible airflow. For this last scenario we are rather limited in what we can do.
The overhead is not that big for just "latest" airflow - we are actually doing it every time when we release a bugfix release already - so 2.3.3 constraints are exactly that when we prepare them.
The question is - if we start adding constraints for "previous" releases of Airflow - should we do it for "latest" stable release only, or for "all versions". Which versions? I think it's a no-go for anything but the latest release due to overhead "per version" it needs.
And to be honest - it does not help a lot to anyone but those who are on the latest version of it is the "latest" only. Plus it is not always possible, we would have to likely manually remove some providers - `pip` resolution is not perfect, as it might get to bactracking cycles, and I am afraid if we "commit" to releasing such constraints, we are going to have a lot of work to solve those. And in some cases (like k8s provider with conflicting kubernets library update) it won't be possible at all.
And it's not really problematic actually. we have roughly 1.5- monthly release cycle for Airlfow bugfixes. So we are usually talking about max 2 weeks gap between those two events. And it will automatically use the latest providers, so I'd rather encourage people to update to latest bugfix release.
Another possibility (hinted in the survey) - maybe even better we could also adapt our release schedule for providers - it does not have to be "exactly monthly" - even now it is "roughly monthly". So we could make sure we release all providers a week (or 1.5 week) before a new bugfix/feature release of Airlfow - making sure that when we release the "latest" it always has "just released" providers as well. That might actually be the easiest and best way to do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org