You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/29 17:45:39 UTC

[GitHub] [airflow] r-richmond opened a new pull request #11950: Add python 3.9 to supported versions

r-richmond opened a new pull request #11950:
URL: https://github.com/apache/airflow/pull/11950


   ---
   Add python 3.9 to the test matrix and supported version list
   
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kaxil commented on pull request #11950: Add python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

kaxil commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-721440733


   Can you please rebased your PR on latest Master since we have applied [Black](https://github.com/apache/airflow/commit/4e8f9cc8d02b29c325b8a5a76b4837671bdf5f68) and [PyUpgrade](https://github.com/apache/airflow/commit/8c42cf1b00c90f0d7f11b8a3a455381de8e003c5) on Master.
   
   It will help if your squash your commits into single commit first so that there are less conflicts.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-730635959


   > Assuming the tests past am I ok to merge?
   
   🤞 
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kaxil commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

kaxil commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-785350226


   +1 -- We can skip the test for the entire-provider for that specific Python version and add some sort of meta information for setup.py for providers -- Since I can see this causing issues in future too


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-730636347


   I also prepared the like DockerHub configuration in the meantime to support "dockerhub" image builds so yeah - I hope it will be ok :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] r-richmond edited a comment on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

r-richmond edited a comment on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-826395881


   @potiuk
   
    On python 3.9 I see you've opened https://github.com/apache/airflow/pull/15515. Should I rebase this PR on master or did you want to take over this pr with 15515?
   
   
   >"Let's use poetry - it will solve all our problems". That was 2 years ago. ...
   Happy to brainstorm on that - but rather that 'use XXX' I would love to see that someone implements it. I.e. "this is how the tool solves this two problems:
   
   Ty for the background; Given that I understand why airflow won't switch off of pip. That said what do you think of instead introducing tests that use poetry or pipenv to ensure that the requirements we have are always valid. i.e. the tests would run lock to make sure no conflicting requirements slip into airflow.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk edited a comment on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-826546862






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] r-richmond edited a comment on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

r-richmond edited a comment on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-798992652


   So you mentioned this before but theres a bootstrap issue of some sort here right?
   running 
   `./breeze build-image --upgrade-to-newer-dependencies --python 3.9 --build-cache-local`
   results in
   `Error response from daemon: manifest for apache/airflow:python3.9-master not found: manifest unknown: manifest unknown`
   Which makes sense since this is the pr trying to add 3.9...
   
   Side question has there been any discussion on switching to another package management solution like [pipenv](https://github.com/pypa/pipenv) or [poetry](https://python-poetry.org). The reason I ask is it feels like there has been a lot of errors around package version conflicts slipping in lately. By using one of the above solutions airflow would have better management over this and it would prevent invalid package management changes from slipping in.
   
   example of one I stumbled across while trying to fool with this was
   `<debug>PackageInfo:</debug> Invalid constraint (googledatastore>=7.0.1,<7.1; python_version < "3.0" ; extra == "gcp") found in apache-beam-2.9.0 dependencies, skipping`
   another example was https://github.com/apache/airflow/issues/13093


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ashb commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

ashb commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-798927739


   > I think @ashb was looking at that at some point in time. Maybe it's even done already.
   
   Not yet, I got as far as creating an issue about it but haven't started working on it. The only "difficult" bit is working out what this means for the DbApiHook method that returns a dataframe.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-719425637


   > We have limited resources on CI and I am concerned that adding a new version may make this problem even worse. If we want to support a new version of Python, we should drop the other version from CI. On the other hand, versions 3.6, 3.7 and 3.8 are the most used versions and we should have tested them well.
   
   @mik-laj -> see above comment. The change I added actually makes it perfectly possible to add a new version of python here. 
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk edited a comment on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-771990816


   Cool. Let's see. I will take a look after we release 2.0.1 :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #11950: Add python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

github-actions[bot] commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-722137312


   [The Workflow run](https://github.com/apache/airflow/actions/runs/346841917) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-730650119


   Ah not... we need constraints :( which are missing yet. Let me build and push those https://raw.githubusercontent.com/apache/airflow/constraints-master/constraints-3.9.txt


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-722240660


   > > Can you please rebased your PR on latest Master since we have applied Black and PyUpgrade on Master.
   > > It will help if your squash your commits into single commit first so that there are less conflicts.
   > 
   > Ok rebased. Last time I believe it failed on mermaid checks and I got lost going down the rabbit hole.
   > 
   > What is the easiest way to ensure that my pr passes all the static checks (black, flake8, mermaid, etc...) ?
   
   Just install pre-commit and run it with `--all-files`. See https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-772440781


   There  is a static check issue with README - I also added 'full-tests-needed' label to make the tests run for all python versions - can you rebase please to rerun them ?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] r-richmond edited a comment on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

r-richmond edited a comment on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-748679045


   Looks like the job was cancelled? 
   Also both urls were the same; was that intentional?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-748906769


   > Even latest version of pyarrow does not yet have python 3.9 version released so we have to wait until they release next version with py3.9: https://pypi.org/project/pyarrow/2.0.0/#files
   
   And we have to wait for snowflake python library improvements because they might introduce some problems.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] r-richmond commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

r-richmond commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-826395881


   On python 3.9 I see you've opened https://github.com/apache/airflow/pull/15515. Should I rebase this PR on master or did you want to take over this pr with 15515?
   
   
   >"Let's use poetry - it will solve all our problems". That was 2 years ago. ...
   Happy to brainstorm on that - but rather that 'use XXX' I would love to see that someone implements it. I.e. "this is how the tool solves this two problems:
   
   Ty for the background; Given that I understand why airflow won't switch off of pip. That said what do you think of instead introducing tests that use poetry or pipenv to ensure that the requirements we have are always valid. i.e. the tests would run lock to make sure no conflicting requirements slip into airflow.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-730637482


   The problem is, that it is a bit "experimental. and we need to add some "leap of faith"  - the images are built in master and 3.9 build is not there I am afraid. But what I can do, I can push your image to my personal repo as master and see if it works (I am going to do it right now).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] r-richmond commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

r-richmond commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-772658843


   Rebased on master and re-ran pre-commit. pre-commit made a change to the README.md which hopefully is the issue that the check was complaining about.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] r-richmond commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

r-richmond commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-771948343


   I saw that pyarrow published a [3.9 wheel for their latest version 3.x](https://pypi.org/project/pyarrow/3.0.0/#files) so I rebased this pr against master in case its ready to test again. Although looking above it seems that the pyarrow restriction may still cause problems?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-785402468


   This is not only tests - those are dependencies. Which would mean that if we want to skip particular provider you'd have to skip it entirely
    Right now we have a 'blessed' set of dependencies for our CI build. We should exclude all.such 'bad operators' when it happens from even that. We already do that for Apache  Beam. But i think for Snowflake it will be solved next week so issue solved 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] r-richmond commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

r-richmond commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-798992652


   So you mentioned this before but theres a bootstrap issue of some sort here right?
   running 
   `./breeze build-image --upgrade-to-newer-dependencies --python 3.9 --build-cache-local`
   results in
   `Error response from daemon: manifest for apache/airflow:python3.9-master not found: manifest unknown: manifest unknown`
   
   Side question has there been any discussion on switching to another package management solution like [pipenv](https://github.com/pypa/pipenv) or [poetry](https://python-poetry.org). The reason I ask is it feels like there has been a lot of errors around package version conflicts slipping in lately. By using one of the above solutions airflow would have better management over this and it would prevent invalid package management changes from slipping in.
   
   example of one I stumbled across while trying to fool with this was
   `<debug>PackageInfo:</debug> Invalid constraint (googledatastore>=7.0.1,<7.1; python_version < "3.0" ; extra == "gcp") found in apache-beam-2.9.0 dependencies, skipping`
   another example was https://github.com/apache/airflow/issues/13093


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-728843139


   Look good to me. I will just approve it, get the "full-tests-needed" label


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] mik-laj commented on pull request #11950: Add python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

mik-laj commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-719076086


   We have limited resources on CI and I am concerned that adding a new version may make this problem even worse.  If we want to support a new version of Python, we should drop the other version from CI. On the other hand, versions 3.6, 3.7 and 3.8 are the most used versions and we should have tested them well.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kaxil edited a comment on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

kaxil edited a comment on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-785350226


   +1 -- We can skip the test for the entire-provider for that specific Python version and add some sort of meta information for setup.py for providers -- Since I can see this causing issues in future too.
   
   (or remove it from Airflow repo entirely)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk edited a comment on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-730636347


   I also prepared the DockerHub configuration in the meantime to support "dockerhub" image builds so yeah - I hope it will be ok :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #11950: Add python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

github-actions[bot] commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-718952101


   [The Workflow run](https://github.com/apache/airflow/actions/runs/336484847) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] r-richmond commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

r-richmond commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-798811746


   Not so subtle bump... Any updates here? Looks like snowflake has [released](https://github.com/snowflakedb/snowflake-connector-python/releases) a couple versions so that blocker would be gone now if I've followed everything correctly.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-730649560


   And you can see all 4 images being built here: https://github.com/potiuk/airflow/actions/runs/373143617 - 3.9 should be a little slower (12 minutes to build rather than 4) because it is not from the cache, but it should hopefully work :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kaxil commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

kaxil commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-785408009


   +1


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-785325458


   The library is planned to be relased next week (I know it from a good source) :). So looks like Python 3.9 support soon :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-826546862

For Python 3.9 I think I will simply go with the #15515 as in order to merge it there will be one more "tricky" point which is to make sure that Python 3.9 is also built in the "build images" workflow. Once this will be merged, I think it will have all the changes from that PR so I think we will be able to close this one (I will be happy to make you co-author of the #15515 :).

> Ty for the background; Given that I understand why airflow won't switch off of pip. That said what do you think of instead introducing tests that use poetry or pipenv to ensure that the requirements we have are always valid. i.e. the tests would run lock to make sure no conflicting requirements slip into airflow.

PIP 21 (PR to be merged in #15513) already has a working resolver that is detecting conflicts much better and we will continue using it to prevent conflicts once it is merged. So this problem will be solved. I think once teething problems of the new resolver have been solved and we will be able to solve all the current problems (with #15513). I think the new PIP removes any need for any extra tools like poetry or pipenv (and is much more versatile IMHO). @uranusjr maybe you can comment on that one.

The problem we have (and it is also described in https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#airflow-dependencies, Airflow is both library and application. And constraint mechanism allows us to handle it well.

* On one hand we can keep the dependencies open in `setup.py` (so that the users of airflow are not limited and can upgrade dependencies after Airflow is released)
* At the same time, with the --constraint switch and publishing "golden" set of constraints we are able to make sure that users installing released version of Airflow can also install it in the future.

AFAIK (and please correct me if I am wrong) neither Poetry nor PIPEnv bring anything more, actually the workflows they support are limiting. They are more opinionated and allow either the library approach (everything as open as possible) or the application one (everything fixed with poetry/pipenv .lock file). From what I understand, the information about dependency versions from those .lock files is not available in the PyPI after packages are released. You either "bake it" into the package (for application) or strip it (for library). Unlike PIP, as far as I know neither Poetry nor Pyenv has the remote URL for lock-files capability that `--constraint` flag of PIP gives us - we can take "open-dependency" Airflow and point to the "fixed" constraints that we store in an orphaned branch in our GitHub Repo and voila - open-dependency "Airflow" is installed in a repeatable way with "golden" set of dependencies.

However I am really open to add "support" for people who are using Poetry or PIPenv. I know there are many users of poetry/pipenv out there. If you are one of the poetry/pipenv users and could write a nice HowTo - how to take Airflow constraints and install airflow with Poetry and Pipenv I am all ears.

It would have to use somehow the constraint file we have - otherwise it will not work in the future with the open-ended approach of Airlfow dependencies. I looked at it briefly, but I could not find a one-line equivalent of `pip install apache-airflow[extras]==2.0.2 --constraints https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-3.7.txt` that we have for PIP installation.

If this means that we will have to store poetry.lock file in our orphan branch next to constraints, I am more than happy to help with automating it.

Would you be willing to help with that and prepare/maintain such "instructions" ?

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk edited a comment on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-827407628

> Anyways I could write a small guide on using Pipenv with airflow. Whats interesting is that there is nothing special about airflow + pipenv so I don't know how much of it would be novel but if you think it would help I'd be happy to. Where do you think it should go in the repo?

Cool. I think the best thing is about the installation https://github.com/apache/airflow/blob/master/docs/apache-airflow/installation.rst . I believe this is the best place as we are talking mostly about the "user" side. I.e. we are not talking about poetry/pipenv used during development of airflow, but about installing airflow from PyPI (via poetry/pipenv). This is the main place where people get to know how they are installing airflow as users.

There is potentially second doc: https://github.com/apache/airflow/blob/master/INSTALL which is the "low-level" installation from sources (and one that always have to be there because PyPI release is only the "convenience" installation packge - the only "official" release of Airlfow is at https://downloads.apache.org/airflow/ and this INSTALL describes how to install Airflow from the sources. But since this is a "power user" doc, I think leaving PIP as the only installation method is quite OK - the less "freedom" we have there, the better.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-730639774


   Build is running here: https://github.com/potiuk/airflow/actions/runs/373119363


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #11950: Add python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

github-actions[bot] commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-718949977


   [The Workflow run](https://github.com/apache/airflow/actions/runs/336480290) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-730650315


   [First time adding new python version for a loooooong time] :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] r-richmond commented on pull request #11950: Add python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

r-richmond commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-723485711


   >That looks like a transient error. I think you can mark it as "ready for review" and rebase to latest master
   
   Done


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-719424469

> Given python's deprecation schedule I believe it is a reasonable assumption that anything that runs on min & max version will run on the versions between as well.

Unfortunately, this does not work. We've had a number of python 3.6-only problems in the past where things were working for 3.5 and 3.7. Mainly because of making assumptions on sorting dictionary items and similar (implicit) behaviours that developers assumed were working on all python versions.

However, this is not really needed. We've introduced a recent change that all PRs by definition are run on default values of matrix - until they are changing Core of airflow in which case they are automatically marked as "full tests needed" when approved by the committer and only then full matrix of tests will be run - we are testing it right now (but it seems to work pretty well).

After the merge, the full set of tests will be run, so we are going to catch any incompatibilities "outside" the core as well. See https://github.com/apache/airflow/pull/11828#issuecomment-718174087

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-771990816


   Cool. Let's see. I will take a look after we rlease 2.0.1 :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk edited a comment on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-730639774


   Build is running here: https://github.com/potiuk/airflow/runs/1426982529?check_suite_focus=true


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] r-richmond commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

r-richmond commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-784414001


   First off thank you for taking another stab at this. Turns out its a ton of work to get the next version working...
   
   >We are using snowflake connector which has a very narrow requirements for pyarrow -> 0.17.1 is the latest version supported. 
   
   Pardon my ignorance here but is there no way to specify python versions for certain providers?
   i.e. it seems sub optimal to require that every provider support the new python version before airflow can be updated as that could lead to less popular providers blocking the rest of the community.
   
   One historical example of a single package/provider causing longterm python version issues was the snakebite package issue which caused all sorts of python 3 headaches.
   
   > The good news is that I am a Snowflake employee now and will try to make it happen :)
   
   Congrats! That also definitely gives some hope for updating the snowflake connector :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-784542480


   Yep. I think the same. If we cannot make snowflake and beam py3.9 compliant, we should exclude them from community managed providers. I hope it will not happen and this thread is an important input to at least one of the discussions I am going to have about it. SOON


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ashb commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

ashb commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-784540464


   > > i.e. it seems sub optimal to require that every provider support the new python version before airflow can be updated as that could lead to less popular providers blocking the rest of the community.
   > 
   > Yeah. It is suboptimal. And we already had a number of problems because of that. 
   
   The point about all operators running in one env is not @r-richmond's point -- more that _all the other providers_ can't be marked as supporting Py3.9 until every provider does.
   
   If this isn't fixed soon, we should have a way of excluding certain providers from running tests on some python versions, so that we could release 3.9 support for everything else.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] r-richmond commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

r-richmond commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-826395881


   On python 3.9 I see you've opened https://github.com/apache/airflow/pull/15515. Should I rebase this PR on master or did you want to take over this pr with 15515?
   
   
   >"Let's use poetry - it will solve all our problems". That was 2 years ago. ...
   Happy to brainstorm on that - but rather that 'use XXX' I would love to see that someone implements it. I.e. "this is how the tool solves this two problems:
   
   Ty for the background; Given that I understand why airflow won't switch off of pip. That said what do you think of instead introducing tests that use poetry or pipenv to ensure that the requirements we have are always valid. i.e. the tests would run lock to make sure no conflicting requirements slip into airflow.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-798888042

Indeed snowflake is now bumped and I believe it is not a blocker any more.

I built the image recently and unfortunately I think we have another dependency problem - pyarrow/numpy which we are using in a version that has no binary packages built for python 3.9. Possibly that can be fixed by building the older versions of those libraries, but I think ultimately it means that we need to either get rid of the requirments from airflow core (which I believe are still there) or upgrade to newer versions of those. I think @ashb was looking at that at some point in time. Maybe it's even done already.

To be honest (and to get the expectations right) - python 3.9 support is rather low priority, I am looking at it usually somewhere between other tasks when I am waiting for CI etc. So maybe you would like to contintue taking a look at it yourself @r-richmond ? We are dealing now with much higher priority task - speeding the tests for everyone by 2x - 6x times.

I am happy to help if you just try to run it and do some initial investigations. What we really need to achieve is to have a CI image built with this command (--build-cache-local is there so that you can locally iterate on the image without pulling/using caches from remote):

```
./breeze build-image --upgrade-to-newer-dependencies --python 3.9 --build-cache-local
```
So if you would like to try to build it and see if the problems can be - maybe you can try and see if those problems I mentioned can be fixed.

WDYT @r-richmond ?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-748444393


   Image build running here: https://github.com/potiuk/airflow/actions/runs/432157502
   CI build here: https://github.com/potiuk/airflow/actions/runs/432157502


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] r-richmond commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

r-richmond commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-748323782


   Looks like with 2.0 the version of numpy is higher now so @potiuk can you try again?
   
   note: this was with a base version of airflow 2.0 no providers, if that matters.
   
   https://github.com/apache/airflow/blob/master/setup.cfg#L119
   ```
   numpy==1.19.4
     - pandas==1.1.5 [requires: numpy>=1.15.4]
       - apache-airflow==2.0.0 [requires: pandas>=0.17.1,<2.0]
         - apache-airflow-providers-ftp==1.0.0 [requires: apache-airflow>=2.0.0a0]
         - apache-airflow-providers-http==1.0.0 [requires: apache-airflow>=2.0.0a0]
         - apache-airflow-providers-imap==1.0.0 [requires: apache-airflow>=2.0.0a0]
         - apache-airflow-providers-sqlite==1.0.0 [requires: apache-airflow>=2.0.0a0]
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] r-richmond closed pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

r-richmond closed pull request #11950:
URL: https://github.com/apache/airflow/pull/11950


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] r-richmond commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

r-richmond commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-730617300


   >Look good to me. I will just approve it, get the "full-tests-needed" label
   
   Assuming the tests past am I ok to merge?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-748902837


   Unfortunately still failing with Pyarrrow:
   
   ```
   ERROR: Command errored out with exit status 1:
      command: /usr/local/bin/python /usr/local/lib/python3.9/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-ft7nb0h3/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.6.0' wheel cython 'pyarrow>=0.17.0,<0.18.0'
          cwd: None
     Complete output (3432 lines):
     Collecting setuptools>=40.6.0
       Downloading setuptools-51.1.0.post20201221-py3-none-any.whl (2.0 MB)
     Collecting wheel
       Downloading wheel-0.36.2-py2.py3-none-any.whl (35 kB)
     Collecting cython
       Downloading Cython-0.29.21-cp39-cp39-manylinux1_x86_64.whl (1.9 MB)
     Collecting pyarrow<0.18.0,>=0.17.0
       Downloading pyarrow-0.17.1.tar.gz (2.6 MB)
       Installing build dependencies: started
       Installing build dependencies: finished with status 'error'
       ERROR: Command errored out with exit status 1:
        command: /usr/local/bin/python /usr/local/lib/python3.9/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-dsvx64_w/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'cython >= 0.29' 'numpy==1.14.5; python_version<'"'"'3.7'"'"'' 'numpy==1.16.0; python_version>='"'"'3.7'"'"'' setuptools setuptools_scm wheel
            cwd: None
   ```
   
   Even latest version of pyarrow does not yet have python 3.9 version released so we have to wait until they release next version with py3.9:  https://pypi.org/project/pyarrow/2.0.0/#files


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] r-richmond commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

r-richmond commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-748679045


   Looks like the job was cancelled? 
   Also both urls were the same?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-783795633


   Unfortunately, I tried to get the 3.9 build working but to no avail.
   
   We are using snowflake connector which has a  very narrow requirements for pyarrow -> 0.17.1 is the latest version supported. And this version requires numpy == 1.16.0 which does not have python 3.9 version and does not even compile for python 3.9 because it uses tp_print enum which has been removed from Python 3.9 (https://bugs.python.org/issue39361)
   
   ` error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’; did you mean ‘tp_dict’?`
   
   So unless we can upgrade the snowflake connector to use newer pyarrow version, we cannot add python 3.9.
   
   The good news is that I am a Snowflake employee now and will try to make it happen :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-784527287

> i.e. it seems sub optimal to require that every provider support the new python version before airflow can be updated as that could lead to less popular providers blocking the rest of the community.

Yeah. It is suboptimal. And we already had a number of problems because of that. But the way it works currently is that - unforatunately - all providers and core run in one python env and this means they have to share dependencies. Not much we can do now, and while we ha some ideas how to improve that, this is - in most cases - not as big problem as it seems. In most cases we have lower-bound dependencies (rather than upper-bound) and different providers have only small subset of overlapping dependencies.

So there are just a few providers that are problematic - and snowflake is one of them. The benefits of not having to introduce isolation between providers are important - simplicity, debuggability, easy deployment, easy management of envs, upgrades etc. Are important. and we developed the whole CI process to keep the "golden" dependencies up-to-date.

There are really two problematic providers/dependencies:

* apache-beam
* snowflake

If we can help to sort them out, we should be really good - without having to introduce another approach. Taking into account that 3 of Airflow Committers /PMC members are now @ Snowflake - I think we can improve the former. And 3 committers of beam are also with us. so.......

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on pull request #11950: Add python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk edited a comment on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-722240660


   > > Can you please rebased your PR on latest Master since we have applied Black and PyUpgrade on Master.
   > > It will help if your squash your commits into single commit first so that there are less conflicts.
   > 
   > Ok rebased. Last time I believe it failed on mermaid checks and I got lost going down the rabbit hole.
   > 
   > What is the easiest way to ensure that my pr passes all the static checks (black, flake8, mermaid, etc...) ?
   
   Just install pre-commit and run it with `--all-files`. See https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst
   
   * You can run individual pre-commits with 'pre-commit run <pre-commit> --all-files`
   * For all the individual changes pre-commit will automatically run at every commit only for subset of changes when you install it with `pre-commit install`
   * You can also use `breeze`. Breeze has the benefit of using auto-complete so that you do not have to look up the ids of pre-commits. You can run it as `./breeze static-checks <TAB>` and you see the ids of pre-commit. You can also add extra options for example `breeze static-checks mypy -- --all-files` will run checks for mypy for all files.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-723175620


   > Ty for the tips. Looks like pre-commit got me through the mermaid errors however the last run failed with the following and I'm not sure where to go from there.
   
   That looks like a transient error. I think you can mark it as "ready for review" and rebase to latest master
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] r-richmond commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

r-richmond commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-781898374


   > Cool. Let's see. I will take a look after we release 2.0.1 :)
   
   Anything else I can help with here?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk edited a comment on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-826546862

* On one hand we can keep the dependencies open in `setup.py` (so that the users of airflow are not limited and can upgrade dependencies after Airflow is released)
* At the same time, with the `--constraint` switch and publishing "golden" set of constraints we are able to make sure that users installing released version of Airflow can also install it in the future.

AFAIK (and please correct me if I am wrong) neither Poetry nor PIPEnv bring anything more, actually the workflows they support are limiting. They are more opinionated and allow either the library approach (everything as open as possible) or the application one (everything fixed with poetry/pipenv .lock file). From what I understand, the information about dependency versions from those .lock files is not available in the PyPI after packages are released. You either "bake it" into the package (for application) or strip it (for library). Unlike PIP, as far as I know neither Poetry nor Pyenv has the remote URL for lock-files capability that `--constraint` flag of PIP gives us. With Pip we can take "open-dependency" Airflow and point to the "fixed" constraints that we store in an orphaned branch in our GitHub Repo and voila - open-dependency "Airflow" is installed in a repeatable way with "golden" set of dependencies.

If this means that we will have to store poetry.lock file in our orphan branch next to constraints, I am more than happy to help with automating it.

Would you be willing to help with that and prepare/maintain such "instructions" ?

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk edited a comment on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-826546862

* On one hand we can keep the dependencies open in `setup.py` (so that the users of airflow are not limited and can upgrade dependencies after Airflow is released)
* At the same time, with the `--constraint` switch and publishing "golden" set of constraints we are able to make sure that users installing released version of Airflow can also install it in the future.

If this means that we will have to store poetry.lock file in our orphan branch next to constraints, I am more than happy to help with automating it.

Would you be willing to help with that and prepare/maintain such "instructions" ?

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] r-richmond commented on pull request #11950: Add python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

r-richmond commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-722495521


   Ty for the tips. Looks like pre-commit got me through the mermaid errors however the last run failed with the following and I'm not sure where to go from there.
   
   ```
   Run actions/upload-artifact@v2
   With the provided path, there will be 1 file(s) uploaded
   Total size of all the files uploaded is 55330 bytes
   Error: read ECONNRESET
   ```
   
   p.s. I added 3.9 to the supported version for 2+ but did not add it to the 1.10 series. Was that correct or should I add it as supported for 1.10.13?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-731035927


   I think it will just have to wait :(


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] r-richmond commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

r-richmond commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-730637372


   Actually here's a question; as I'm watching these tests go I'm not seeing 3.9 versions. What am I missing?
   
   <img width="882" alt="image" src="https://user-images.githubusercontent.com/9246654/99724150-10b20680-2a68-11eb-9de9-e06d03342c36.png">
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-748443425


   Yep. I will test it happily now :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk edited a comment on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-782240187


   > Anything else I can help with here?
   
   I am taking a look tomorrow :). the daily involvement for Airflow will be quite a bit less for me now after Snowflake acquired us, I have a different day projects now. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-798995664


   
   > results in
   > `Error response from daemon: manifest for apache/airflow:python3.9-master not found: manifest unknown: manifest unknown`
   > Which makes sense since this is the pr trying to add 3.9...
   
   I believe this can be fixed by  setting `SKIP_CHECK_REMOTE_IMAGE` to `true` - there we should skip checking if the remote image exists.
   
   > Side question has there been any discussion on switching to another package management solution like [pipenv](https://github.com/pypa/pipenv) or [poetry](https://python-poetry.org). The reason I ask is it feels like there has been a lot of errors around package version conflicts slipping in lately. By using one of the above solutions airflow would have better management over this and it would prevent invalid package management changes from slipping in.
   
   Oh yeah. we tried all that. And about 2 years ago I wrote the email on the devlist which was pretty much "Let's use poetry - it will solve all our problems". That was 2 years ago.
   
   And those tools were all discarded as they did not work well in our complex case. We even have an entry specific about pip-tools and poetry in here https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#airflow-dependencies together with explanation why they would not work. Here is more context on why those tools do not work: https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#dependency-management
   
   There is nothing magical with what pipenv or poetry does. Both 'poetry' and 'pip-tools'  are rather more opinionated than PIP. And that makes them actually less suitable for Airflow  - airflow is both a library and application and neither pip-tools nor poetry handle the case well). We had lots of discussions about this in the devlist. 
   
   I'd really love to see the solution of the problem with other tools where:
   
   * you do not limit airflow with 'install-requires' whtere you can freely keep a set of few hudred libraries that you can upgrade at will
   * but you can install the latest set of airflow + providers and be sure they are working together in a consistent way (no matter if some transitive dependencies were released recently).
   
   Happy to brainstorm on that - but rather that 'use XXX' I would love to see that someone implements it. I.e. "this is how the tool solves this two problems:
   
   a)  a user wants to install released version of Airlfow (say 2.0.2) with set of providers/extras (for example google/snowflake/amazon) and they got it working with non-conflicting requirements
   b) a user wants to upgrade most dependencies to a newer version as they need to write a DAG that uses newer version of the dependency.
   
   So far poetry/pip-tools can work well with a) but they fail miserably when you add b) to the mix. But I am happy to hear how it can be done :)
   
   
   > 
   > example of one I stumbled across while trying to fool with this was
   > `<debug>PackageInfo:</debug> Invalid constraint (googledatastore>=7.0.1,<7.1; python_version < "3.0" ; extra == "gcp") found in apache-beam-2.9.0 dependencies, skipping`
   > another example was #13093
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-785170731


   I wish I could tell that I helped, but I did not not, however soon the problem will be solved: https://github.com/snowflakedb/snowflake-connector-python/commit/ef70176631d500543187168268c29a7b05311f34#diff-60f61ab7a8d1910d86d9fda2261620314edcae5894d5aaa236b821c7256badd7 
   
   I will double check what are the plans of releasing this version, but once it is, I think we will be able to support 3.9 :)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-730647915


   Added right link -> you will see that all 4 versions of the images are running there.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] r-richmond commented on pull request #11950: Add python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

r-richmond commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-722129002


   >Can you please rebased your PR on latest Master since we have applied Black and PyUpgrade on Master.
   It will help if your squash your commits into single commit first so that there are less conflicts.
   
   Ok rebased. Last time I believe it failed on mermaid checks and I got lost going down the rabbit hole.
   
   What is the easiest way to ensure that my pr passes all the static checks (black, flake8, mermaid, etc...) ?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-771990816


   Cool. Let's see. I will take a look after we rlease 2.0.1 :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-827407628

There is potentially second dpc: https://github.com/apache/airflow/blob/master/INSTALL which is the "low-level" installation from sources (and one that always have to be there because PyPI release is only the "convenience" installation packge - the only "official" release of Airlfow is at https://downloads.apache.org/airflow/ and this INSTALL describes how to install Airflow from the sources. But since this is a "power user" doc, I think leaving PIP as the only installation method is quite OK - the less "freedom" we have there, the better.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ashb commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

ashb commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-731115568


   That reminds me: I want to take a look at removing numpy/pyarrow from _core_ airflow. We don't really need it, and it would make installing easier.
   
   After 2.0


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] r-richmond edited a comment on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

r-richmond edited a comment on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-771948343


   I saw that pyarrow published a [3.9 wheel for their latest version 3.x](https://pypi.org/project/pyarrow/3.0.0/#files) so I rebased this pr against master in case its ready to test again. Although looking above it seems that the pyarrow version restriction may still cause problems?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-826546862

If this means that we will have to store poetry.lock file in our orphan branch next to constraints, I am more than happy to help with automating it.

Would you be willing to help with that and prepare/maintain such "instructions" ?

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] r-richmond commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

r-richmond commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-771948343


   I saw that pyarrow published a [3.9 wheel for their latest version 3.x](https://pypi.org/project/pyarrow/3.0.0/#files) so I rebased this pr against master in case its ready to test again. Although looking above it seems that the pyarrow restriction may still cause problems?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

github-actions[bot] commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-772189787


   [The Workflow run](https://github.com/apache/airflow/actions/runs/532547961) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] r-richmond commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

r-richmond commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-827197437


   >For Python 3.9 I think I will simply go with the #15515 as in order to merge it there will be one more "tricky" point which is to make sure that Python 3.9 is also built in the "build images" workflow. Once this will be merged, I think it will have all the changes from that PR so I think we will be able to close this one (I will be happy to make you co-author of the #15515 :).
   
   In that case I'll close this PR out. Thanks for tackling the extra effort for 3.9. As far as co-author goes I've already my Contributor label from other PRS so no need to add me as a co-author for that one. Besides you did a lot of work in this pr as well.
   
   >PIP 21 (PR to be merged in #15513) already has a working resolver that is detecting conflicts much better and we will continue using it to prevent conflicts once it is merged
   
   Oh I'm not familiar with the new resolver. As long as it can detect conflicts this would address my feedback / concern.
   
   >AFAIK (and please correct me if I am wrong) neither Poetry nor PIPEnv bring anything more, actually the workflows they support are limiting. They are more opinionated and allow either the library approach (everything as open as possible) or the application one (everything fixed with poetry/pipenv .lock file)
   
   >However I am really open to add "support" for people who are using Poetry or PIPenv. I know there are many users of poetry/pipenv out there. If you are one of the poetry/pipenv users and could write a nice HowTo - how to take Airflow constraints and install airflow with Poetry and Pipenv I am all ears.
   
   So I'm no expert and my point of view is very heavy from the Application side. So from that point of view the big thing pipenv does is take open-ended requirements and spit out an explicit list of things installed. The nice thing about this is you can pin versions in the Pipfile and leave others open which is much more flexible than a normal frozen requirements. (And of course the big thing reproducible builds). The only thing fragile here is that pipenv and poetry won't let you lock if an invalid set of requirements slip into airflow, Historically this has happened several times https://github.com/apache/airflow/issues/13093 https://github.com/apache/airflow/issues/14145 https://github.com/apache/airflow/issues/13558  with airflow but if as you said above the new Pip resolver detects these we should be good.
   
   Anyways I could write a small guide on using Pipenv with airflow. Whats interesting is that there is nothing special about airflow + pipenv so I don't know how much of it would be novel but if you think it would help I'd be happy to. Where do you think it should go in the repo?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] r-richmond edited a comment on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

r-richmond edited a comment on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-826395881


   @potiuk
   
    On python 3.9 I see you've opened https://github.com/apache/airflow/pull/15515. Should I rebase this PR on master or did you want to take over this pr with 15515?
   
   
   >"Let's use poetry - it will solve all our problems". That was 2 years ago. ...
   Happy to brainstorm on that - but rather that 'use XXX' I would love to see that someone implements it. I.e. "this is how the tool solves this two problems:
   
   Ty for the background; Given that I understand why airflow won't switch off of pip. That said what do you think of instead introducing tests that use poetry or pipenv to ensure that the requirements we have are always valid. i.e. the tests would run lock to make sure no conflicting requirements slip into airflow.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-748866753


   Yeah - that was it :) https://github.com/potiuk/airflow/runs/1586436706 I will have to build and push the image manually first :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk edited a comment on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-827407628

There is potentially second doc: https://github.com/apache/airflow/blob/master/INSTALL which is the "low-level" installation from sources. This one that always have to be there because PyPI release is only the "convenience" installation packge - the only "official" release of Airlfow is at https://downloads.apache.org/airflow/ and this INSTALL describes how to install Airflow from the sources. But since this is a "power user" doc, I think leaving PIP as the only installation method is quite OK - the less "freedom" we have there, the better.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] r-richmond commented on pull request #11950: Add python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

r-richmond commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-719166728


   So currently this pr adds 3.9 to the test matrix which increases it by 33%. If CI resources are a major concern what if instead I modified this pr so that the test matrix only includes the earliest and latest versions (3.6 & 3.9)
   
   That would result in a reduction of CI resources by 33% instead of increase of 33%. Letting tests run faster for all.
   
   Given python's deprecation schedule I believe it is a reasonable assumption that anything that runs on min & max version will run on the versions between as well.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on pull request #11950: Add python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk edited a comment on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-722240660


   > > Can you please rebased your PR on latest Master since we have applied Black and PyUpgrade on Master.
   > > It will help if your squash your commits into single commit first so that there are less conflicts.
   > 
   > Ok rebased. Last time I believe it failed on mermaid checks and I got lost going down the rabbit hole.
   > 
   > What is the easiest way to ensure that my pr passes all the static checks (black, flake8, mermaid, etc...) ?
   
   Just install pre-commit and run it with `--all-files`. See https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst
   
   * You can run all checks with `pre-commit run --all-files` -> this will take about 10-15 minutes depends on the speed of your computer and network and whether you have breeze image recently built
   * You can run individual pre-commits with `pre-commit run <pre-commit> --all-files`
   * For all the individual changes pre-commit will automatically run at every commit only for subset of changes when you install it with `pre-commit install`
   * You can also use `breeze`. Breeze has the benefit of using auto-complete so that you do not have to look up the ids of pre-commits. You can run it as `./breeze static-checks <TAB>` and you see the ids of pre-commit. You can also add extra options for example `breeze static-checks mypy -- --all-files` will run checks for mypy for all files.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk edited a comment on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-748866753


   Yeah - that was a bad copy&paste :) https://github.com/potiuk/airflow/runs/1586436706 I will have to build and push the image manually first :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on pull request #11950: Add python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk edited a comment on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-722240660


   > > Can you please rebased your PR on latest Master since we have applied Black and PyUpgrade on Master.
   > > It will help if your squash your commits into single commit first so that there are less conflicts.
   > 
   > Ok rebased. Last time I believe it failed on mermaid checks and I got lost going down the rabbit hole.
   > 
   > What is the easiest way to ensure that my pr passes all the static checks (black, flake8, mermaid, etc...) ?
   
   Just install pre-commit and run it with `--all-files`. See https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst
   
   * You can run individual pre-commits with `pre-commit run <pre-commit> --all-files`
   * For all the individual changes pre-commit will automatically run at every commit only for subset of changes when you install it with `pre-commit install`
   * You can also use `breeze`. Breeze has the benefit of using auto-complete so that you do not have to look up the ids of pre-commits. You can run it as `./breeze static-checks <TAB>` and you see the ids of pre-commit. You can also add extra options for example `breeze static-checks mypy -- --all-files` will run checks for mypy for all files.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ashb edited a comment on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

ashb edited a comment on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-798927739


   > I think @ashb was looking at that at some point in time. Maybe it's even done already.
   
   Not yet, I got as far as creating #12500 about it but haven't started working on it. The only "difficult" bit is working out what this means for the DbApiHook method that returns a dataframe.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-782240187


   > Anything else I can help with here?
   
   I am taking a look tomorrow :). the daily involvement for Airflow will be quite a bit less for me now :(. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kaxil commented on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

kaxil commented on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-782299028


   Unfortunately, we are dealing with CI issues again, hoping that it gets better tomorrow and we can take further actions here.
   
   Thank you for your patience @r-richmond 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

potiuk edited a comment on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-771990816


   Cool. Let's see. I will take a look after we release 2.0.1 :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] r-richmond edited a comment on pull request #11950: Add Python 3.9 to supported versions

Posted by GitBox <gi...@apache.org>.

r-richmond edited a comment on pull request #11950:
URL: https://github.com/apache/airflow/pull/11950#issuecomment-771948343


   I saw that pyarrow published a [3.9 wheel for their latest version 3.x](https://pypi.org/project/pyarrow/3.0.0/#files) so I rebased this pr against master in case its ready to test again. Although looking above it seems that the pyarrow version restriction may still cause problems?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org