You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2022/11/24 07:37:53 UTC
[GitHub] [superset] EugeneTorap opened a new pull request, #22217: chore: Support Python 3.11
EugeneTorap opened a new pull request, #22217:
URL: https://github.com/apache/superset/pull/22217
### SUMMARY
Add Python 3.11 support for superset which gives huge performance improvements (25% is the average number claimed)
Bump next libs:
- cffi==1.15.1
- numpy==1.23.5
- pandas==1.5.2
- pyarrow==10.0.1
For Postgres also bump `psycopg2-binary` to 2.9.5
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
<!--- Skip this if not applicable -->
### TESTING INSTRUCTIONS
<!--- Required! What steps can be taken to manually verify the changes? -->
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] EugeneTorap commented on a diff in pull request #22217: chore: Support Python 3.11
Posted by GitBox <gi...@apache.org>.
EugeneTorap commented on code in PR #22217:
URL: https://github.com/apache/superset/pull/22217#discussion_r1038349522
##########
requirements/base.txt:
##########
@@ -177,7 +177,7 @@ prison==0.2.1
# via flask-appbuilder
prompt-toolkit==3.0.28
# via click-repl
-pyarrow==6.0.1
+pyarrow>=9.0.0, <11 # Warning! python 3.11 requires pyarrow==10.0.1 but some testing libs require pyarrow<10
Review Comment:
@villebro Or we can wait when db-dtypes 1.0.5 will be released
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] villebro commented on a diff in pull request #22217: chore: Support Python 3.11
Posted by GitBox <gi...@apache.org>.
villebro commented on code in PR #22217:
URL: https://github.com/apache/superset/pull/22217#discussion_r1038364385
##########
requirements/base.txt:
##########
@@ -177,7 +177,7 @@ prison==0.2.1
# via flask-appbuilder
prompt-toolkit==3.0.28
# via click-repl
-pyarrow==6.0.1
+pyarrow>=9.0.0, <11 # Warning! python 3.11 requires pyarrow==10.0.1 but some testing libs require pyarrow<10
Review Comment:
@EugeneTorap let's wait for 1.0.5 to be released, the Pandas team tend to release fairly frequently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] EugeneTorap commented on a diff in pull request #22217: chore: Support Python 3.11
Posted by GitBox <gi...@apache.org>.
EugeneTorap commented on code in PR #22217:
URL: https://github.com/apache/superset/pull/22217#discussion_r1038346913
##########
requirements/base.txt:
##########
@@ -177,7 +177,7 @@ prison==0.2.1
# via flask-appbuilder
prompt-toolkit==3.0.28
# via click-repl
-pyarrow==6.0.1
+pyarrow>=9.0.0, <11 # Warning! python 3.11 requires pyarrow==10.0.1 but some testing libs require pyarrow<10
Review Comment:
I fix it in next PR when google guys will release 1.0.5 for `db-dtypes`
https://github.com/googleapis/python-db-dtypes-pandas/issues/161
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] villebro commented on pull request #22217: chore: Support Python 3.11
Posted by GitBox <gi...@apache.org>.
villebro commented on PR #22217:
URL: https://github.com/apache/superset/pull/22217#issuecomment-1336045168
On this topic, I happened to come across this interesting Lex Friedman interview with Guido van Rossum about what the main optimizations are in 3.11: https://youtu.be/TLhRuZ9cJWc . Looking forward to JIT in a future version.. 😛
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] villebro merged pull request #22217: chore: Support Python 3.11
Posted by GitBox <gi...@apache.org>.
villebro merged PR #22217:
URL: https://github.com/apache/superset/pull/22217
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] villebro commented on a diff in pull request #22217: chore: Support Python 3.11
Posted by GitBox <gi...@apache.org>.
villebro commented on code in PR #22217:
URL: https://github.com/apache/superset/pull/22217#discussion_r1038743319
##########
setup.py:
##########
@@ -126,6 +126,7 @@ def get_git_sha() -> str:
"athena": ["pyathena[pandas]>=2, <3"],
"aurora-data-api": ["preset-sqlalchemy-aurora-data-api>=0.2.8,<0.3"],
"bigquery": [
+ "db-dtypes @ git+https://git@github.com/googleapis/python-db-dtypes-pandas.git@388e082a47d9515a14e20ffd87705c71712087ab#egg=db-dtypes",
Review Comment:
In a hurry I see 😆
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] villebro commented on a diff in pull request #22217: chore: Support Python 3.11
Posted by GitBox <gi...@apache.org>.
villebro commented on code in PR #22217:
URL: https://github.com/apache/superset/pull/22217#discussion_r1038343440
##########
requirements/base.txt:
##########
@@ -177,7 +177,7 @@ prison==0.2.1
# via flask-appbuilder
prompt-toolkit==3.0.28
# via click-repl
-pyarrow==6.0.1
+pyarrow>=9.0.0, <11 # Warning! python 3.11 requires pyarrow==10.0.1 but some testing libs require pyarrow<10
Review Comment:
I don't think the lock file should have ranges. Let's pin this to the latest 9.x version, and maybe document somewhere that Python 3.11 requires installing >= 10.0.1? What were the libs that require PyArrow < 10?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] EugeneTorap commented on a diff in pull request #22217: chore: Support Python 3.11
Posted by GitBox <gi...@apache.org>.
EugeneTorap commented on code in PR #22217:
URL: https://github.com/apache/superset/pull/22217#discussion_r1038346913
##########
requirements/base.txt:
##########
@@ -177,7 +177,7 @@ prison==0.2.1
# via flask-appbuilder
prompt-toolkit==3.0.28
# via click-repl
-pyarrow==6.0.1
+pyarrow>=9.0.0, <11 # Warning! python 3.11 requires pyarrow==10.0.1 but some testing libs require pyarrow<10
Review Comment:
I will fix it in next PR when google guys will release 1.0.5 for `db-dtypes`
https://github.com/googleapis/python-db-dtypes-pandas/issues/161
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] codecov[bot] commented on pull request #22217: chore: Support Python 3.11
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #22217:
URL: https://github.com/apache/superset/pull/22217#issuecomment-1335378154
# [Codecov](https://codecov.io/gh/apache/superset/pull/22217?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#22217](https://codecov.io/gh/apache/superset/pull/22217?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4f3227e) into [master](https://codecov.io/gh/apache/superset/commit/7bc5f04368ecf1feb82799d51903438d0b2fe145?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7bc5f04) will **decrease** coverage by `11.35%`.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## master #22217 +/- ##
===========================================
- Coverage 66.89% 55.53% -11.36%
===========================================
Files 1847 1847
Lines 70333 70333
Branches 7702 7702
===========================================
- Hits 47047 39058 -7989
- Misses 21288 29277 +7989
Partials 1998 1998
```
| Flag | Coverage Δ | |
|---|---|---|
| hive | `?` | |
| mysql | `?` | |
| postgres | `?` | |
| presto | `52.45% <ø> (ø)` | |
| python | `57.43% <ø> (-23.88%)` | :arrow_down: |
| sqlite | `?` | |
| unit | `50.86% <ø> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/superset/pull/22217?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [superset/utils/dashboard\_import\_export.py](https://codecov.io/gh/apache/superset/pull/22217/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdXRpbHMvZGFzaGJvYXJkX2ltcG9ydF9leHBvcnQucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [superset/tags/core.py](https://codecov.io/gh/apache/superset/pull/22217/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdGFncy9jb3JlLnB5) | `4.54% <0.00%> (-95.46%)` | :arrow_down: |
| [superset/key\_value/commands/update.py](https://codecov.io/gh/apache/superset/pull/22217/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQva2V5X3ZhbHVlL2NvbW1hbmRzL3VwZGF0ZS5weQ==) | `0.00% <0.00%> (-90.91%)` | :arrow_down: |
| [superset/key\_value/commands/delete.py](https://codecov.io/gh/apache/superset/pull/22217/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQva2V5X3ZhbHVlL2NvbW1hbmRzL2RlbGV0ZS5weQ==) | `0.00% <0.00%> (-87.88%)` | :arrow_down: |
| [superset/db\_engines/hive.py](https://codecov.io/gh/apache/superset/pull/22217/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lcy9oaXZlLnB5) | `0.00% <0.00%> (-85.19%)` | :arrow_down: |
| [superset/key\_value/commands/delete\_expired.py](https://codecov.io/gh/apache/superset/pull/22217/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQva2V5X3ZhbHVlL2NvbW1hbmRzL2RlbGV0ZV9leHBpcmVkLnB5) | `0.00% <0.00%> (-84.00%)` | :arrow_down: |
| [superset/dashboards/commands/importers/v0.py](https://codecov.io/gh/apache/superset/pull/22217/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGFzaGJvYXJkcy9jb21tYW5kcy9pbXBvcnRlcnMvdjAucHk=) | `15.62% <0.00%> (-76.25%)` | :arrow_down: |
| [superset/datasets/commands/create.py](https://codecov.io/gh/apache/superset/pull/22217/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGF0YXNldHMvY29tbWFuZHMvY3JlYXRlLnB5) | `30.61% <0.00%> (-69.39%)` | :arrow_down: |
| [superset/datasets/commands/update.py](https://codecov.io/gh/apache/superset/pull/22217/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGF0YXNldHMvY29tbWFuZHMvdXBkYXRlLnB5) | `25.00% <0.00%> (-69.05%)` | :arrow_down: |
| [superset/datasets/commands/importers/v0.py](https://codecov.io/gh/apache/superset/pull/22217/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGF0YXNldHMvY29tbWFuZHMvaW1wb3J0ZXJzL3YwLnB5) | `24.03% <0.00%> (-69.00%)` | :arrow_down: |
| ... and [282 more](https://codecov.io/gh/apache/superset/pull/22217/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] EugeneTorap commented on pull request #22217: chore: Support Python 3.11
Posted by GitBox <gi...@apache.org>.
EugeneTorap commented on PR #22217:
URL: https://github.com/apache/superset/pull/22217#issuecomment-1335416288
Use `pyarrow>=9.0.0, <11` in requirements/base.txt because `python 3.11` requires `pyarrow>=10.0.1` but some testing libs require `pyarrow<10`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] EugeneTorap commented on a diff in pull request #22217: chore: Support Python 3.11
Posted by GitBox <gi...@apache.org>.
EugeneTorap commented on code in PR #22217:
URL: https://github.com/apache/superset/pull/22217#discussion_r1038352736
##########
requirements/base.txt:
##########
@@ -177,7 +177,7 @@ prison==0.2.1
# via flask-appbuilder
prompt-toolkit==3.0.28
# via click-repl
-pyarrow==6.0.1
+pyarrow>=9.0.0, <11 # Warning! python 3.11 requires pyarrow==10.0.1 but some testing libs require pyarrow<10
Review Comment:
@betodealmeida @zhaoyongjie What do you think about it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org