You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2022/09/08 18:31:48 UTC
[GitHub] [superset] justinpark opened a new issue, #21385: [SIP-88] Proposal for non-blocking sqlLab persistence
justinpark opened a new issue, #21385:
URL: https://github.com/apache/superset/issues/21385
## [SIP-88] Proposal for async sqlLab persistence
### Motivation
At Airbnb we have a performance (lag) issue using `SQLLAB_BACKEND_PERSISTENCE` feature.
Since `SQLLAB_BACKEND_PERSISTENCE` designed in a blocking way (as shown in the following), user can be stuck while `tab_stat` table in busy state. (even unable to run a query while the `tabstateview` endpoint timed out)
https://github.com/apache/superset/blob/8c2719b11cebe451c5ba68193cf8dd51e4dce8e3/superset-frontend/src/SqlLab/actions/sqlLab.js#L614-L625
In order to improve this lag issue, we'd like to have a non-blocking mechanism for this db syncing workflow. i.e. like g-doc auto saving. moreover we'd like to design a hybrid solution (i.e. keep using redux-localstorage and save same snaphot in remote) to keep the advantage (offline saving) of using local storage.
### Proposed Change
The proposed solution is simple. We would like to keep using redux-localstorage as well as the current localStorage mechanism (as SQLLAB_BACKEND_PERSISTENCE turned OFF). Like QueryAutoSync does, we would like to build a component that observes the changes of queryEditor and then periodically sync the updated queryEditor snapshot from redux state using existing `/tabstateview/` api.
For the payload wise, we'd like to use `extra_json` in `tab_stat` table to store the snapshot(by json format) of queryEditor as stored in localStorage (rather than mapping each value in a proper table column like sql, label, query_limit).
As existing persistence feature does, we will design similar hydration flow in frontend to initialize the sqlLab state from the backend. With this json blob it will simply convert to json and then merge with the existing local state.
### New or Changed Public Interfaces
We will introduce new `SQLLAB_BACKEND_PERSISTENCE_ASYNC` FLAG and need to pass the `tab_state_ids` value including `extra_json` in this case.
### New dependencies
I do not for-see any new dependencies.
### Migration Plan and Compatibility
I do not for-see any migration required.
### Rejected Alternatives
None.
to: @etr2460 @ktmud @john-bodley
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] ktmud commented on issue #21385: [SIP-88] Proposal for non-blocking SQL Lab persistence
Posted by GitBox <gi...@apache.org>.
ktmud commented on issue #21385:
URL: https://github.com/apache/superset/issues/21385#issuecomment-1241120843
I like this idea. Can you elaborate more on which states will be pulled to local storage and which states will be kept exclusively on remote? I'd imagine we probably shouldn't keep even a temporary local copy of big payloads such as query results.
<img width="485" alt="image" src="https://user-images.githubusercontent.com/335541/189205958-31f8737e-ad17-4331-af85-a8d24f4b7061.png">
For states that can be loaded fresh upon page load--such as `tables` and `databases`, they should also be skipped for localStorage in this async approach.
<img width="425" alt="image" src="https://user-images.githubusercontent.com/335541/189204207-ef8235a4-e979-4b17-b644-7fab0cadf901.png">
There are also some duplicates in saved states, too, that we may want to clean up:
<img width="474" alt="image" src="https://user-images.githubusercontent.com/335541/189204753-f8853ae5-5fdb-493d-9c32-00bc55b33e0b.png">
I also noticed we are saving previous queries of current active tab even though they may never be used again. Should be able to save a lot of space if we just clean them up.
---
To summarize, I think it'd be useful to start cleaning up current saved states when `SQLLAB_BACKEND_PERSISTENCE` is off, before migrating to this hybrid approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] justinpark commented on issue #21385: [SIP-88] Proposal for non-blocking SQL Lab persistence
Posted by GitBox <gi...@apache.org>.
justinpark commented on issue #21385:
URL: https://github.com/apache/superset/issues/21385#issuecomment-1241136749
> I like this idea. Can you elaborate more on which states will be pulled to local storage and which states will be kept exclusively on remote? I'd imagine we probably shouldn't keep even a temporary local copy of big payloads such as query results.
>
Good point. There's a helper function for this and planning to reuse this function to trim the unnecessary items.
https://github.com/apache/superset/blob/5f9f657805a5559de6f06c647a33f278aadc32db/superset-frontend/src/SqlLab/utils/reduxStateToLocalStorageHelper.js#L56-L60
> To summarize, I think it'd be useful to start cleaning up current saved states
100% agree. we should clean up the following keys and save only necessary
https://github.com/apache/superset/blob/5f9f657805a5559de6f06c647a33f278aadc32db/superset-frontend/src/SqlLab/utils/reduxStateToLocalStorageHelper.js#L21-L37
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] michael-s-molina commented on issue #21385: [SIP-93] Proposal for non-blocking SQL Lab persistence
Posted by GitBox <gi...@apache.org>.
michael-s-molina commented on issue #21385:
URL: https://github.com/apache/superset/issues/21385#issuecomment-1317975891
Interesting idea @justinpark . I wonder if we should provide both sync and async persistence options or just provide the async version as the evolution of the feature. If it's the latter, we could reuse the same feature flag given that we're planning to clean up the current state, which will modify the existing feature anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] mistercrunch commented on issue #21385: [SIP-93] Proposal for non-blocking SQL Lab persistence
Posted by GitBox <gi...@apache.org>.
mistercrunch commented on issue #21385:
URL: https://github.com/apache/superset/issues/21385#issuecomment-1325520167
Took a look today, this is comprehensive, seems reasonable and I don't think there's anything controversial here.
+1
Though I'd love to get @betodealmeida 's blessing since I believe he implemented the original `SQLLAB_BACKEND_PERSISTENCE`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-93] Proposal for non-blocking SQL Lab persistence [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina closed issue #21385: [SIP-93] Proposal for non-blocking SQL Lab persistence
URL: https://github.com/apache/superset/issues/21385
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] geido commented on issue #21385: [SIP-93] Proposal for non-blocking SQL Lab persistence
Posted by GitBox <gi...@apache.org>.
geido commented on issue #21385:
URL: https://github.com/apache/superset/issues/21385#issuecomment-1323794484
Nice proposal @justinpark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-93] Proposal for non-blocking SQL Lab persistence [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on issue #21385:
URL: https://github.com/apache/superset/issues/21385#issuecomment-1808604530
Closing the SIP as it was approved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] justinpark commented on issue #21385: [SIP-93] Proposal for non-blocking SQL Lab persistence
Posted by GitBox <gi...@apache.org>.
justinpark commented on issue #21385:
URL: https://github.com/apache/superset/issues/21385#issuecomment-1332768481
> If it's the latter, we could reuse the same feature flag given that we're planning to clean up the current state, which will modify the existing feature anyway.
@michael-s-molina
You're right but I suggested new feature flag to reduce the failure/loss of existing persist functionality due to the error by new integration.
I hope to make new async persistence option first and then replace/consolidate with the same persist feature flag name once no running bugs found from our cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org