[GitHub] [superset] giftig opened a new issue, #24343: SqlaTable query hangs on list tables API call with large number of tables

["giftig (via GitHub)" <gi...@apache.org>]

giftig opened a new issue, #24343:
URL: https://github.com/apache/superset/issues/24343

   An inefficient `SqlaTable` query causes the table list endpoint to hang when you have a large number of tables in the schema.
   
   #### How to reproduce the bug
   
   1. Populate a few thousand datasources for tables in a schema you want to list
   2. Go to SQL Lab and try to get the table list for your selected schema
   3. Observe the call times out as the API does not respond in a timely manner
   
   ### Expected results
   
   Efficient return of table list from this endpoint
   
   ### Actual results
   
   API call hangs
   
   ### Environment
   
   (please complete the following information):
   
   - browser type and version: N/A
   - superset version: `2.1.0`
   
   ### Checklist
   
   Make sure to follow these steps before submitting your issue - thank you!
   
   - [x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
   - [x] I have reproduced the issue with at least the latest released version of superset.
   - [x] I have checked the issue tracker for the same issue and I haven't found one similar.
   
   ### Additional context
   
   I've gotten the bottom of this issue and will raise a PR. Essentially the call to `SqlaTable` to retrieve associated table info and populate the `extra` field is extremely inefficient with a large number of queries, and that's because of the eager loading of the `columns` and `metrics` relationships: the direct query being called is relatively fine even with thousands of rows, but when I inspected with echo on, sqlalchemy was performing an additional ~41 queries, including several outer joins with these additional tables and several queries like `WHERE table_id IN (...thousands of IDs)`.
   
   This can be avoided by using the `load_only` and `lazyload` options on this query.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org

[GitHub] [superset] giftig closed issue #24343: SqlaTable query hangs on list tables API call with large number of tables

["giftig (via GitHub)" <gi...@apache.org>]

giftig closed issue #24343: SqlaTable query hangs on list tables API call with large number of tables
URL: https://github.com/apache/superset/issues/24343


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org