You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2022/04/28 21:05:45 UTC
[GitHub] [superset] john-bodley opened a new pull request, #19887: fix(hive): Workaround for Python 3.9 s3 transfer issue
john-bodley opened a new pull request, #19887:
URL: https://github.com/apache/superset/pull/19887
<!---
Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/
Example:
fix(dashboard): load charts correctly
-->
### SUMMARY
This PR remedies an issue we (Airbnb) were facing after upgrading to Python 3.9. Per https://github.com/boto/s3transfer/issues/197#issuecomment-1016026027 there seems to be an s3 threading with Python 3.9+ causing the transfer to fail. Disabling threading seems to mitigate the issue. Note it's unclear what the potential performance impact is, especially when uploading large files.
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
<!--- Skip this if not applicable -->
### TESTING INSTRUCTIONS
CI and verified within an Airbnb environment.
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] codecov[bot] commented on pull request #19887: fix(hive): Workaround for Python 3.9 s3 transfer issue
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #19887:
URL: https://github.com/apache/superset/pull/19887#issuecomment-1112656177
# [Codecov](https://codecov.io/gh/apache/superset/pull/19887?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#19887](https://codecov.io/gh/apache/superset/pull/19887?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (56742ed) into [master](https://codecov.io/gh/apache/superset/commit/ea6753d27df81f57d691593a7ec131bcfaf1e7ea?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ea6753d) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.
```diff
@@ Coverage Diff @@
## master #19887 +/- ##
=======================================
Coverage 66.52% 66.52%
=======================================
Files 1714 1714
Lines 65032 65033 +1
Branches 6717 6717
=======================================
+ Hits 43260 43261 +1
Misses 20065 20065
Partials 1707 1707
```
| Flag | Coverage Δ | |
|---|---|---|
| hive | `52.91% <0.00%> (-0.01%)` | :arrow_down: |
| javascript | `51.25% <ø> (ø)` | |
| mysql | `81.92% <100.00%> (+<0.01%)` | :arrow_up: |
| postgres | `81.96% <100.00%> (+<0.01%)` | :arrow_up: |
| presto | `52.76% <0.00%> (-0.01%)` | :arrow_down: |
| python | `82.39% <100.00%> (+<0.01%)` | :arrow_up: |
| sqlite | `81.73% <100.00%> (+<0.01%)` | :arrow_up: |
| unit | `47.99% <0.00%> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/superset/pull/19887?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [superset/db\_engine\_specs/hive.py](https://codecov.io/gh/apache/superset/pull/19887/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2hpdmUucHk=) | `85.87% <100.00%> (+0.05%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/superset/pull/19887?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/superset/pull/19887?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [ea6753d...56742ed](https://codecov.io/gh/apache/superset/pull/19887?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] etr2460 commented on a diff in pull request #19887: fix(hive): Workaround for Python 3.9 s3 transfer issue
Posted by GitBox <gi...@apache.org>.
etr2460 commented on code in PR #19887:
URL: https://github.com/apache/superset/pull/19887#discussion_r864445256
##########
superset/db_engine_specs/hive.py:
##########
@@ -80,6 +81,7 @@ def upload_to_s3(filename: str, upload_prefix: str, table: Table) -> str:
filename,
bucket_path,
os.path.join(upload_prefix, table.table, os.path.basename(filename)),
+ Config=TransferConfig(use_threads=False),
Review Comment:
```suggestion
# Disabling threading because it breaks python 3.9
Config=TransferConfig(use_threads=False),
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] john-bodley commented on a diff in pull request #19887: fix(hive): Workaround for Python 3.9 s3 transfer issue
Posted by GitBox <gi...@apache.org>.
john-bodley commented on code in PR #19887:
URL: https://github.com/apache/superset/pull/19887#discussion_r864963139
##########
superset/db_engine_specs/hive.py:
##########
@@ -80,6 +81,7 @@ def upload_to_s3(filename: str, upload_prefix: str, table: Table) -> str:
filename,
bucket_path,
os.path.join(upload_prefix, table.table, os.path.basename(filename)),
+ Config=TransferConfig(use_threads=False),
Review Comment:
@etr2460 I inlined the comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] john-bodley merged pull request #19887: fix(hive): Workaround for Python 3.9 s3 transfer issue
Posted by GitBox <gi...@apache.org>.
john-bodley merged PR #19887:
URL: https://github.com/apache/superset/pull/19887
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org