You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/04/25 06:44:42 UTC
[GitHub] [airflow] potiuk opened a new pull request #15515: Add python 3.9
potiuk opened a new pull request #15515:
URL: https://github.com/apache/airflow/pull/15515
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#issuecomment-867530296
All good EXCEPT the "no-member" pylint error in non-touched code (again).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] r-richmond commented on pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
r-richmond commented on pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#issuecomment-855469492
@potiuk in case you missed it; it looks like thrift_sasl released [0.4.3](https://github.com/cloudera/thrift_sasl/releases) which as far as I can tell was the version we were waiting for.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#discussion_r656172949
##########
File path: setup.cfg
##########
@@ -106,7 +107,7 @@ install_requires =
graphviz>=0.12
gunicorn>=19.5.0
httpx
- importlib_metadata~=1.7;python_version<"3.9" # We could work with 3.1, but argparse needs <2
+ importlib_metadata~=3.1;python_version<"3.9"
Review comment:
I actually changed it to >=1.7 now to handle current constraint files, so we should automatically get latest compatible version.
BTW. Should not we install `importlib-metadata` regardless of python version ? This is what jira package does anyway and it seems it gets updated more frequently than Python 3.9 is. We might - at some point in time start relying on some features there. Seems that importlib_metadata 4.4 for example is only available in 3.10 (https://github.com/python/importlib_metadata).
Maybe we should just remove `;python<3.9`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] r-richmond commented on pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
r-richmond commented on pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#issuecomment-873323918
Is python 3.9 not supported on 2.1.1 (i.e. we missed the cutoff) or was that an oversight on the docs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#discussion_r656249335
##########
File path: setup.cfg
##########
@@ -106,7 +107,7 @@ install_requires =
graphviz>=0.12
gunicorn>=19.5.0
httpx
- importlib_metadata~=1.7;python_version<"3.9" # We could work with 3.1, but argparse needs <2
+ importlib_metadata~=3.1;python_version<"3.9"
Review comment:
Mostly following the principle that "we don't need any of the modern features here, and less things installed = less to worry about"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#issuecomment-868511050
Just one approve away from having Python 3.9 support in main :D.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#issuecomment-867519663
Looks like it gets green all-over: https://github.com/potiuk/airflow/actions/runs/967459087
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#issuecomment-860264240
Bad news @r-richmond. While thrift-sasl is already good-to-go ...... seems that we had one more hidden dependency on sasl :(
@uranusjr warned me several times that extras in PIP are totally insane, and make it next to impossible to build a reasonable dependency graph - and I must concur with it now..... In our case `pyhive[hive] ` makes us still download and try to compile `sasl` (which has no chance to succeed as `sasl` support for 3.9 even if working for alpha release in April has been apparently abandoned).
I added comment to the relevant issue in PyHive:
https://github.com/dropbox/PyHive/issues/380#issuecomment-860263187
Upvotes, encouraging comments are welcome :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#discussion_r656145406
##########
File path: setup.cfg
##########
@@ -106,7 +107,7 @@ install_requires =
graphviz>=0.12
gunicorn>=19.5.0
httpx
- importlib_metadata~=1.7;python_version<"3.9" # We could work with 3.1, but argparse needs <2
+ importlib_metadata~=3.1;python_version<"3.9"
Review comment:
I think you meant argcomplete; it has a importlib-metadata pin (and pip would get this wrong before the resolver upgrade). argcomplete is compatible with `importlib-metadata<5` now though, so this should work. (Maybe we should look into importlib-metadata 4.x compatibility as well.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] r-richmond commented on pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
r-richmond commented on pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#issuecomment-876004070
Do we know what the next release version will be / Any rough timelines for 2.2?
(context: I'm trying to figure out how long we'd have to wait if we waited for 2.2 for python 3.9 vs if its worth just doing the workarounds you mentioned above).
Relevant milestone info that I can view
* [2.1.2 milestone](https://github.com/apache/airflow/milestone/37) has 6 completed items and 0 open items with a due date of July 10th.
* [2.1.3 milestone](https://github.com/apache/airflow/milestone/40) has 24 completed items and 17 open items with no due date
* [2.2.0 milestone](https://github.com/apache/airflow/milestone/33) has 23 completed items and 32 open items with no due date
p.s. thanks again for tackling this update; I'm really excited about a couple quality of life things in 3.9.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#issuecomment-855743622
> @potiuk in case you missed it; it looks like thrift_sasl released [0.4.3](https://github.com/cloudera/thrift_sasl/releases) which as far as [I can tell](https://github.com/cloudera/thrift_sasl/issues/36#issuecomment-828739117) was the version we were waiting for.
Cool. I will look at this this week :). 🤞
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#issuecomment-873369903
Also this is very much documented in https://github.com/apache/airflow#requirements that 3.9 is for now only available officially for development version (which is 2.2.0dev currently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#issuecomment-826268991
cc: @r-richmond - This will not YET work. But after we migrate to PIP 21 fully with #15515 (I hope in a day or two) and after Apache Beam releases their 2.29.0 (Release candidate is out there as of 16th of April) we might finally be able to pull that one off! In Apache Beam we have pyarrow limitation moved to <4.0.0 and Pyarrow 3.0.0 is already python 3.9 compatible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#issuecomment-827835592
All Right apache-beam 2.29 released! Let's give it a try :).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#issuecomment-826268991
cc: @r-richmond - This will not YET work. But after we migrate to PIP 21 fully with #15515 (I hope in a day or two) and after Apache Beam releases their 2.29.0 (Release candidate is out there as of 16th of April https://pypi.org/project/apache-beam/2.29.0rc1/) we might finally be able to pull that one off! In Apache Beam we have pyarrow limitation moved to <4.0.0 and Pyarrow 3.0.0 is already python 3.9 compatible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#discussion_r656248362
##########
File path: setup.cfg
##########
@@ -106,7 +107,7 @@ install_requires =
graphviz>=0.12
gunicorn>=19.5.0
httpx
- importlib_metadata~=1.7;python_version<"3.9" # We could work with 3.1, but argparse needs <2
+ importlib_metadata~=3.1;python_version<"3.9"
Review comment:
Oh yes, argcomplete sounds right
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#discussion_r656172949
##########
File path: setup.cfg
##########
@@ -106,7 +107,7 @@ install_requires =
graphviz>=0.12
gunicorn>=19.5.0
httpx
- importlib_metadata~=1.7;python_version<"3.9" # We could work with 3.1, but argparse needs <2
+ importlib_metadata~=3.1;python_version<"3.9"
Review comment:
I actually changed it to >1.7 now to handle current constraint files, so we should automatically get latest compatible version.
BTW. Should not we install `importlib-metadata` regardless of python version ? This is what jira package does anyway and it seems it gets updated more frequently than Python 3.9 is. We might - at some point in time start relying on some features there. Seems that importlib_metadata 4.4 for example is only available in 3.10 (https://github.com/python/importlib_metadata).
Maybe we should just remove `;python<3.9`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#issuecomment-826360553
BTW. Seems that voting for the RC candidate for Beam has been completed and there is just mechanics of the release, so it is more than likely tha the 2.29.0 will be out today/tomorrow and we will be able to add Python 3.9 finally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#discussion_r656176219
##########
File path: setup.cfg
##########
@@ -106,7 +107,7 @@ install_requires =
graphviz>=0.12
gunicorn>=19.5.0
httpx
- importlib_metadata~=1.7;python_version<"3.9" # We could work with 3.1, but argparse needs <2
+ importlib_metadata~=3.1;python_version<"3.9"
Review comment:
The marker was introduced in #12862. I don’t have an opinion either way (both have their advantages and supported by different people).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#issuecomment-868357537
Shall we?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#issuecomment-876306072
@r-richmond Airflow 2.2 will be roughly 3 months form 2.1.0, so ~end of August.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#issuecomment-868555735
👯 🕺
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#issuecomment-873369903
Also this is very much documented in https://github.com/apache/airflow#requirements that 3.9 is for now only available officially for development version (which is 2.2.0dev currently). This is similar to MSSQL support which is currently only available in `dev`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#issuecomment-826268991
cc: @r-richmond - This will not YET work. But after we migrate to PIP 21 fully with #15515 (I hope in a day or two) and after Apache Beam releases their 2.29.0 (Release candidate is out there as of 16th of April https://pypi.org/project/apache-beam/2.29.0rc1/) we might finally be able to pull that one off! In Apache Beam we have pyarrow limitation moved to <4.0.0 and Pyarrow 3.0.0 is already python 3.9 compatible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#issuecomment-826268991
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#discussion_r656145406
##########
File path: setup.cfg
##########
@@ -106,7 +107,7 @@ install_requires =
graphviz>=0.12
gunicorn>=19.5.0
httpx
- importlib_metadata~=1.7;python_version<"3.9" # We could work with 3.1, but argparse needs <2
+ importlib_metadata~=3.1;python_version<"3.9"
Review comment:
I think you meant argcomplete; it has a importlib-metadata pin (and pip would get this wrong before the resolver upgrade). argcomplete was updated to be compatible with `importlib-metadata<5` now though, so this should work. (Maybe we should look into importlib-metadata 4.x compatibility as well.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a change in pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on a change in pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#discussion_r621686741
##########
File path: scripts/ci/selective_ci_checks.sh
##########
@@ -115,9 +115,9 @@ function output_all_basic_variables() {
initialization::ga_output default-helm-version "${HELM_VERSION}"
if [[ ${FULL_TESTS_NEEDED_LABEL} == "true" ]]; then
- initialization::ga_output postgres-exclude '[{ "python-version": "3.6" }]'
- initialization::ga_output mysql-exclude '[{ "python-version": "3.7" }]'
- initialization::ga_output sqlite-exclude '[{ "python-version": "3.8" }]'
+ initialization::ga_output postgres-exclude '[{ "python-version": "3.6" }, { "python-version": "3.9" }]'
+ initialization::ga_output mysql-exclude '[{ "python-version": "3.7" }, { "python-version": "3.9" }]'
+ initialization::ga_output sqlite-exclude '[{ "python-version": "3.7" }, { "python-version": "3.8" }]'
Review comment:
Curious, why is 3.9 being excluded for postgres/mysql and 3.7 now being excluded for sqlite?
##########
File path: CI.rst
##########
@@ -57,7 +57,7 @@ Container Registry used as cache
For the CI builds of our we are using Container Registry to store results of the "Build Image" workflow
and pass it to the "CI Build" workflow.
-Currently in master version of Airflow we run tests in 3 different versions of Python (3.6, 3.7, 3.8)
+Currently in master version of Airflow we run tests in 3 different versions of Python (3.6, 3.7, 3.8, 3.9)
Review comment:
```suggestion
Currently in master version of Airflow we run tests in 4 different versions of Python (3.6, 3.7, 3.8, 3.9)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#discussion_r656085018
##########
File path: setup.cfg
##########
@@ -106,7 +107,7 @@ install_requires =
graphviz>=0.12
gunicorn>=19.5.0
httpx
- importlib_metadata~=1.7;python_version<"3.9" # We could work with 3.1, but argparse needs <2
+ importlib_metadata~=3.1;python_version<"3.9"
Review comment:
Hmmmm, I'm afraid I can't remember, but sphinx-argparse doesn't use importlib etc.
If it works and passes tests :+1:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#issuecomment-876309041
In the meantime @r-richmond I released Python 3.9 compatble elasticsearch package and I have done some extra work and cherry-picked the Python3.9 changes to `2-1-stable` branch: https://github.com/apache/airflow/pull/16883.
I also generated constraints.
We are just about to release 2.1.2 and If the tests will run fine, we will merge it and then Python 3.9 will be easy to install for 2.1.2. Not sure if we mark it as officially available for 2.1.2, but maybe you will be able to help with testing it ?
We can push an image (I just build it, it builds fine), Also constraints are already there:
https://github.com/apache/airflow/commit/924d038313a08b571a49dd6582a7143402cce31f
So once we tag it with 2.1.2, you will be able to install it both via docker and in venv using the same constraints pattern as for other versions.
If you confirm it works fine for you we might even make it "officialy available" in 2.1.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#discussion_r621946098
##########
File path: scripts/ci/selective_ci_checks.sh
##########
@@ -115,9 +115,9 @@ function output_all_basic_variables() {
initialization::ga_output default-helm-version "${HELM_VERSION}"
if [[ ${FULL_TESTS_NEEDED_LABEL} == "true" ]]; then
- initialization::ga_output postgres-exclude '[{ "python-version": "3.6" }]'
- initialization::ga_output mysql-exclude '[{ "python-version": "3.7" }]'
- initialization::ga_output sqlite-exclude '[{ "python-version": "3.8" }]'
+ initialization::ga_output postgres-exclude '[{ "python-version": "3.6" }, { "python-version": "3.9" }]'
+ initialization::ga_output mysql-exclude '[{ "python-version": "3.7" }, { "python-version": "3.9" }]'
+ initialization::ga_output sqlite-exclude '[{ "python-version": "3.7" }, { "python-version": "3.8" }]'
Review comment:
Somewhat arbitrary. We just want to save build/test time so we do not run full.matrix of tests. This is a way to test some 'representative' combinations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#discussion_r656175477
##########
File path: setup.cfg
##########
@@ -106,7 +107,7 @@ install_requires =
graphviz>=0.12
gunicorn>=19.5.0
httpx
- importlib_metadata~=1.7;python_version<"3.9" # We could work with 3.1, but argparse needs <2
+ importlib_metadata~=3.1;python_version<"3.9"
Review comment:
It also makes less 'moving parts' - with < 3.9 we need to handle (and actually should test) 3 cases:
* < 3.9 + importlib_metadata
* > = 3.9
* > = 3.9 +importlib_metadata
By removing <3.9 we have less variability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] r-richmond edited a comment on pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
r-richmond edited a comment on pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#issuecomment-855469492
@potiuk in case you missed it; it looks like thrift_sasl released [0.4.3](https://github.com/cloudera/thrift_sasl/releases) which as far as [I can tell](https://github.com/cloudera/thrift_sasl/issues/36#issuecomment-828739117) was the version we were waiting for.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#issuecomment-876404294
Seems like we are on a good path to have support for Python 3.9 in 2.1.2 @r-richmond, the change is merged there, constraints are updated and the tests generally pass, so I think even officially we might want to state (once 2.1.2 is released) that we have Python 3.9 support for it already. It would be great if you could test the release candidate once it is out :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] millin commented on pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
millin commented on pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#issuecomment-873358020
Please backport python 3.9 support to 2.1.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#discussion_r656175477
##########
File path: setup.cfg
##########
@@ -106,7 +107,7 @@ install_requires =
graphviz>=0.12
gunicorn>=19.5.0
httpx
- importlib_metadata~=1.7;python_version<"3.9" # We could work with 3.1, but argparse needs <2
+ importlib_metadata~=3.1;python_version<"3.9"
Review comment:
It also makes less 'moving parts' - with < 3.9 we need to handle (and actually should test) 3 cases:
* < 3.9 + importlib_metadata
* >= 3.9
* >= 3.9 +importlib_metadata
By removing <3.9 we have less variability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] r-richmond commented on pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
r-richmond commented on pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#issuecomment-860303357
Well that is disappointing but certainly outside airflow's control. Thank you for taking another stab at it.
That said given how painful this has been I opened https://github.com/apache/airflow/issues/16426 which was related / based off of https://github.com/apache/airflow/pull/11950#issuecomment-784540464
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#issuecomment-873369599
Yes. It's not "officially" supported yet. It's been added to main and not backported to 2.1.1. The 2.1 line is released from v2-1-stable branch not from main and we try to avoid excessive cherry-picking between the branches (unless it is necessary).
The 3.9 support is three things really:
* CI tests - those make sense to add/fix for main only
* Constraint files
* Docker images
As a workaround you can still install airflow or build a docker image for python 3.9 with Airflow 2.1.1 and it will probably work fine.
This is quite possible (though not guaranteed to work) using the main version of constraints. The full guarantees we get only when we run the full test suite (which only happens in main for Python 3.9).
Here is how you can build Airflow 2.1.1 Python 3.9 image (you can run it from main airlfow sources):
```
docker build . \
--build-arg PYTHON_BASE_IMAGE="python:3.9-slim-buster" \
--build-arg AIRFLOW_VERSION="2.1.1" \
--build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-main" \
--tag "my-custom-3-9-airflow-image:2.1.1"
```
Similarly installing airflow locally:
```
AIRFLOW_VERSION=2.1.1
PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-${PYTHON_VERSION}.txt"
pip install --upgrade "apache-airflow[postgres,google]==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"
```
Also both Docker and local installation can be run with locally modified version of constraints if you find some conflicts, happy to help if you have problems.
I think as you are eager users that might be perfect opportunity for you to try it and let us know if you see any problems before we release 2.2 which will have full 3.9 support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#discussion_r656074871
##########
File path: setup.cfg
##########
@@ -106,7 +107,7 @@ install_requires =
graphviz>=0.12
gunicorn>=19.5.0
httpx
- importlib_metadata~=1.7;python_version<"3.9" # We could work with 3.1, but argparse needs <2
+ importlib_metadata~=3.1;python_version<"3.9"
Review comment:
@ashb - I guess the argparse limitation is gone already ? I am not quite sure what "argparse" probiem it was (the only argparse which is not-built in is the sphinx-argparse).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#issuecomment-867494354
@ashb @r-richmond - I think this one is ready to merge (finally!). I cancelled it - because it will continue failing with missing Py3.9 image until it is merged, but I am running the same PR in my fork here: https://github.com/potiuk/airflow/actions/runs/967459087 and I succeeded last time (I had a small typo in selective checks that made overall build fail here: https://github.com/potiuk/airflow/actions/runs/965717898).
Looking forward to reviews :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#discussion_r656175477
##########
File path: setup.cfg
##########
@@ -106,7 +107,7 @@ install_requires =
graphviz>=0.12
gunicorn>=19.5.0
httpx
- importlib_metadata~=1.7;python_version<"3.9" # We could work with 3.1, but argparse needs <2
+ importlib_metadata~=3.1;python_version<"3.9"
Review comment:
It also makes less 'moving parts' - with < 3.9 we need to handle (and actually should test) 3 cases:
* < 3.9 + importlib_metadata
* >= 3.9
* >= 3.9 +importlib_metadata
By removing <3.9 we have less variability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#issuecomment-873369903
Also this is very much documented in https://github.com/apache/airflow#requirements that 3.9 is for now only available officially for development version (which is 2.2.0dev currently).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#issuecomment-827918375
Not yet :(. We also have sasl dependency which is non-py3.9 compliant.
But there is a release coming for sasl and thrift-sasl soon to address it too. I asked for some timeline information: https://github.com/cloudera/thrift_sasl/issues/36#issuecomment-827917133
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #15515: Add python 3.9
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #15515:
URL: https://github.com/apache/airflow/pull/15515#issuecomment-876404294
Seems like we are on a good path to have support Python 3.9 for 2.1.2 @r-richmond, the change is merged there, constraints are updated and the tests generally pass, so I think even officially we might want to state (once 2.1.2 is released) that we have Python 3.9 support for it already. It would be great if you could test the release candidate once it is out :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org