You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/03/07 13:48:05 UTC
[GitHub] [airflow] potiuk opened a new pull request #22050: Add Python 3.10 support
potiuk opened a new pull request #22050:
URL: https://github.com/apache/airflow/pull/22050
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22050: Add Python 3.10 support
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22050:
URL: https://github.com/apache/airflow/pull/22050#issuecomment-1065938743
BTW. @alexott - this might be one reason why I **COULD** speed up release of the Databricks provider :P
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22050: Add Python 3.10 support
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22050:
URL: https://github.com/apache/airflow/pull/22050#issuecomment-1066110326
> I've looked into 2.0.0b1 changes, it should be relatively simple to port, only may need to add new options. But it still uses `collections.Iterable` instead of `collections.abc.Iterable`... Will follow up with developers tomorrow
Thanks!
So I will make an exclusion for db tests for now for 3.10 as this might take a bit of time I guess (and we can revert the exclusion and make a follow-up release of databricks provider with lower-bound on the version).
Alternatively a quick fix and release of 1.0.2 of connector might solve it much faster, I think with lower risk of problems for the current provider ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #22050: Add Python 3.10 support
Posted by GitBox <gi...@apache.org>.
potiuk closed pull request #22050:
URL: https://github.com/apache/airflow/pull/22050
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22050: Add Python 3.10 support
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22050:
URL: https://github.com/apache/airflow/pull/22050#issuecomment-1061586058
> @potiuk Do I have to make changes in Breeze2 to support Python 3.10?
No need. I made a "bulk" update here and I also updated `Breeze2` :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #22050: Add Python 3.10 support
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #22050:
URL: https://github.com/apache/airflow/pull/22050#issuecomment-1066110326
> I've looked into 2.0.0b1 changes, it should be relatively simple to port, only may need to add new options. But it still uses `collections.Iterable` instead of `collections.abc.Iterable`... Will follow up with developers tomorrow
Thanks!
So I will make an exclusion for db tests for now for 3.10 as this might take a bit of time I guess (and we can revert the exclusion and make a follow-up release of databricks provider with lower-bound on the version).
Alternatively a quick fix and release of 1.0.2 of connector (just collections.Iterable -> collections.abc.Iterable) might solve it much faster, I think with lower risk of problems for the current provider ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #22050: Add Python 3.10 support
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #22050:
URL: https://github.com/apache/airflow/pull/22050
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #22050: Add Python 3.10 support
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #22050:
URL: https://github.com/apache/airflow/pull/22050#issuecomment-1061468024
@potiuk Do I have to make changes in Breeze2 to support Python 3.10?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexott commented on pull request #22050: Add Python 3.10 support
Posted by GitBox <gi...@apache.org>.
alexott commented on pull request #22050:
URL: https://github.com/apache/airflow/pull/22050#issuecomment-1066104920
I've looked into 2.0.0b1 changes, it should be relatively simple to port, only may need to add new options. But it still uses `collections.Iterable` instead of `collections.abc.Iterable`... Will follow up with developers tomorrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #22050: Add Python 3.10 support
Posted by GitBox <gi...@apache.org>.
potiuk closed pull request #22050:
URL: https://github.com/apache/airflow/pull/22050
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexott commented on pull request #22050: Add Python 3.10 support
Posted by GitBox <gi...@apache.org>.
alexott commented on pull request #22050:
URL: https://github.com/apache/airflow/pull/22050#issuecomment-1065934809
@potiuk let me raise it with the dev team... Source code is in the pypi itself right now
@NiallEgan - is it possible to fix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #22050: Add Python 3.10 support
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #22050:
URL: https://github.com/apache/airflow/pull/22050#issuecomment-1065568832
We might need to reimplement a small issue in cgroupspy - one of the few last remaining issues for Python 3.10 - see https://github.com/cloudsigma/cgroupspy/issues/13
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #22050: Add Python 3.10 support
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #22050:
URL: https://github.com/apache/airflow/pull/22050#issuecomment-1060776621
Should we remove user warning?
https://github.com/apache/airflow/blob/6db9b0029b98611dd748638ab4834a2275e4fa48/airflow/__main__.py#L38-L43
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22050: Add Python 3.10 support
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22050:
URL: https://github.com/apache/airflow/pull/22050#issuecomment-1065568832
We might need to reimplement a small issue in cgroupspy - the last remaining "test" issues for Python 3.10 - see https://github.com/cloudsigma/cgroupspy/issues/13
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22050: Add Python 3.10 support
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22050:
URL: https://github.com/apache/airflow/pull/22050#issuecomment-1065933341
Hey @alexott - we seem to have a blocker for Python 3.10 enabling in databicks-sql-connector (one of the last few blockers):
```
self = <databricks.sql.common.ParamEscaper object at 0x7fe81c6dd6c0>
item = ['file1', 'file2', 'file3']
def escape_item(self, item):
if item is None:
return 'NULL'
elif isinstance(item, (int, float)):
return self.escape_number(item)
elif isinstance(item, basestring):
return self.escape_string(item)
> elif isinstance(item, collections.Iterable):
E AttributeError: module 'collections' has no attribute 'Iterable'
```
https://github.com/apache/airflow/runs/5523057543?check_suite_focus=true#step:8:16781
This makes the connector unusable on Python 3.10 because finally Python 3.10 dropped the deprecated collections.abc aliases (in this case it should colllections.abc.Iterable). I cannot find the sources and status of the databricks library for that (maybe i do not know where to look for it). I see that there is a 2.0.0b1 release and I wonder if:
a) it has the collections problem fixed
b) will we be able to move to the 2.0.0 version (currently we see <2.0.0 in databricks).
This is really one of the last blockers for us to move to Python 3.10. I could potentially disable databricks providers and tests for Python 3.10 but if this will be fixed quickly, maybe we do not have to do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22050: Add Python 3.10 support
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22050:
URL: https://github.com/apache/airflow/pull/22050#issuecomment-1060772134
In order to generate constraints, I need to release a new wave of providers to PyPI now. And We should be good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22050: Add Python 3.10 support
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22050:
URL: https://github.com/apache/airflow/pull/22050#issuecomment-1060706761
I think we should be close to get Python 3.10 support. Let's see
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22050: Add Python 3.10 support
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22050:
URL: https://github.com/apache/airflow/pull/22050#issuecomment-1060785215
> Should we remove user warning?
>
> https://github.com/apache/airflow/blob/6db9b0029b98611dd748638ab4834a2275e4fa48/airflow/__main__.py#L38-L43
Yep. That too :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #22050: Add Python 3.10 support
Posted by GitBox <gi...@apache.org>.
potiuk closed pull request #22050:
URL: https://github.com/apache/airflow/pull/22050
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22050: Add Python 3.10 support
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22050:
URL: https://github.com/apache/airflow/pull/22050#issuecomment-1065940163
BTW. @alexott @NiallEgan - I looked at 2.0.0b1 code, and it is very different from 1.0.1 - so I think even if it will be fixed in 2.0.0b1, this might mean some provider code changes to adapt. Don't know the code though to know for sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexott commented on pull request #22050: Add Python 3.10 support
Posted by GitBox <gi...@apache.org>.
alexott commented on pull request #22050:
URL: https://github.com/apache/airflow/pull/22050#issuecomment-1065940476
I’ll look tomorrow into a new version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexott commented on pull request #22050: Add Python 3.10 support
Posted by GitBox <gi...@apache.org>.
alexott commented on pull request #22050:
URL: https://github.com/apache/airflow/pull/22050#issuecomment-1066110604
Maybe, I'm planning to dig into it this evening to see how much changes is required for 2.0.0...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22050: Add Python 3.10 support
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #22050:
URL: https://github.com/apache/airflow/pull/22050#issuecomment-1074406782
Final PR to get Python 3.10 support !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org