You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "giftig (via GitHub)" <gi...@apache.org> on 2023/06/09 14:23:32 UTC
[GitHub] [superset] giftig opened a new pull request, #24344: fix(sqllab): Inefficient SqlaTable query (#24343)
giftig opened a new pull request, #24344:
URL: https://github.com/apache/superset/pull/24344
Use a combination of lazyload and load_only to ensure sqlalchemy doesn't eagerly join to additional tables, which was happening in a very inefficient way for a large number of tables and resulting in this endpoint never returning (in addition to high db load)
### SUMMARY
<!--- Describe the change below, including rationale and design decisions -->
### TESTING INSTRUCTIONS
1. Populate a few thousand datasources for tables in a schema you want to list
2. Go to SQL Lab and try to get the table list for your selected schema
3. Observe the call times out as the API responds in a timely manner
More info on the associated issue: https://github.com/apache/superset/issues/24343
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [x] Has associated issue: https://github.com/apache/superset/issues/24343
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] john-bodley commented on pull request #24344: fix(sqllab): Inefficient SqlaTable query (#24343)
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on PR #24344:
URL: https://github.com/apache/superset/pull/24344#issuecomment-1586377285
Thanks @giftig for the change. It seems this endpoint is deprecated and thus I was wondering whether you could make said change to the replacement API endpoint?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] villebro merged pull request #24344: fix(sqllab): Inefficient SqlaTable query (#24343)
Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro merged PR #24344:
URL: https://github.com/apache/superset/pull/24344
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] villebro commented on pull request #24344: fix(sqllab): Inefficient SqlaTable query (#24343)
Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro commented on PR #24344:
URL: https://github.com/apache/superset/pull/24344#issuecomment-1586781510
> Thanks @giftig for the change. It seems this endpoint is deprecated (and slated for deletion in #24342) and thus I was wondering whether you could make said change to the replacement API endpoint?
@john-bodley as this is fixing a serious perf regression affecting master branch right now, I suggest we defer the API endpoint move to a follow-up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] codecov[bot] commented on pull request #24344: fix(sqllab): Inefficient SqlaTable query (#24343)
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #24344:
URL: https://github.com/apache/superset/pull/24344#issuecomment-1584679094
## [Codecov](https://app.codecov.io/gh/apache/superset/pull/24344?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#24344](https://app.codecov.io/gh/apache/superset/pull/24344?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (dc9084e) into [master](https://app.codecov.io/gh/apache/superset/commit/bdb8bbef32cf9f782b6a7dc7a93c1ad737712536?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (bdb8bbe) will **decrease** coverage by `11.26%`.
> The diff coverage is `100.00%`.
> :exclamation: Current head dc9084e differs from pull request most recent head 02040e5. Consider uploading reports for the commit 02040e5 to get more accurate results
```diff
@@ Coverage Diff @@
## master #24344 +/- ##
===========================================
- Coverage 69.08% 57.82% -11.26%
===========================================
Files 1905 1905
Lines 74848 74849 +1
Branches 8108 8108
===========================================
- Hits 51705 43280 -8425
- Misses 21031 29457 +8426
Partials 2112 2112
```
| Flag | Coverage Δ | |
|---|---|---|
| hive | `?` | |
| mysql | `?` | |
| postgres | `?` | |
| presto | `53.59% <100.00%> (+<0.01%)` | :arrow_up: |
| python | `60.14% <100.00%> (-23.13%)` | :arrow_down: |
| sqlite | `?` | |
| unit | `54.08% <100.00%> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://app.codecov.io/gh/apache/superset/pull/24344?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [superset/views/core.py](https://app.codecov.io/gh/apache/superset/pull/24344?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvdmlld3MvY29yZS5weQ==) | `36.52% <100.00%> (-38.76%)` | :arrow_down: |
... and [302 files with indirect coverage changes](https://app.codecov.io/gh/apache/superset/pull/24344/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org