You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "justinpark (via GitHub)" <gi...@apache.org> on 2023/03/07 20:35:50 UTC
[GitHub] [superset] justinpark opened a new pull request, #23302: fix(sqllab): empty large query results from localStorage
justinpark opened a new pull request, #23302:
URL: https://github.com/apache/superset/pull/23302
### SUMMARY
At Airbnb, there's a bunch query results including `JSON` data which can easily reaches to the max storage size.
<img width="1439" alt="Screenshot 2023-03-07 at 12 20 04 PM" src="https://user-images.githubusercontent.com/1392866/223543151-fde935de-cfbd-4d01-8d8b-7c36a20b980c.png">
The current empty-out logic only cleans based on the query age, so the huge size of the query result can be stored in localStorage until it expires. However, this long-lived large size query result can not only be disrupted by continuous warning messages, but also slow down SQLlab due to the cost of JSON.stringify on the large JSON object.
This commit adds the extra empty-out logic for the large size query results.
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
After:
https://user-images.githubusercontent.com/1392866/223546035-f15f705f-0ddf-4b41-b4fe-39416d2a39ec.mov
Before:
https://user-images.githubusercontent.com/1392866/223546043-324ada68-918d-4fd0-9849-d54bd5b91f19.mov
### TESTING INSTRUCTIONS
- specs
- go to sqllab with `SQLLAB_BACKEND_PERSISTENCE` off
- run a query that contains large-size JSON column results
- keep using sqllab and check the speed
### ADDITIONAL INFORMATION
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
@ktmud cc: @john-bodley
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] justinpark commented on a diff in pull request #23302: fix(sqllab): empty large query results from localStorage
Posted by "justinpark (via GitHub)" <gi...@apache.org>.
justinpark commented on code in PR #23302:
URL: https://github.com/apache/superset/pull/23302#discussion_r1132696602
##########
superset-frontend/src/SqlLab/utils/emptyQueryResults.test.js:
##########
@@ -45,6 +50,38 @@ describe('reduxStateToLocalStorageHelper', () => {
expect(emptiedQuery[id].results).toEqual({});
});
+ it('should empty query.results if query,.results size is greater than LOCALSTORAGE_MAX_QUERY_RESULTS_KB', () => {
+ const reasonableSizeQuery = {
+ ...queries[0],
+ startDttm: Date.now(),
+ results: { data: [{ a: 1 }] },
+ };
+ const largeQuery = {
+ ...queries[1],
+ startDttm: Date.now(),
+ results: {
+ data: [
+ {
+ test: 1123123123123,
+ stringValue:
+ 'Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industrys standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum',
Review Comment:
I updated with this suggestion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] codecov[bot] commented on pull request #23302: fix(sqllab): empty large query results from localStorage
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #23302:
URL: https://github.com/apache/superset/pull/23302#issuecomment-1458900320
# [Codecov](https://codecov.io/gh/apache/superset/pull/23302?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#23302](https://codecov.io/gh/apache/superset/pull/23302?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (be77dc9) into [master](https://codecov.io/gh/apache/superset/commit/f7e765227009b9df399a07b215f723d2f9b1710f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f7e7652) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.
> :exclamation: Current head be77dc9 differs from pull request most recent head 4d0de19. Consider uploading reports for the commit 4d0de19 to get more accurate results
```diff
@@ Coverage Diff @@
## master #23302 +/- ##
=======================================
Coverage 67.51% 67.51%
=======================================
Files 1909 1909
Lines 73488 73491 +3
Branches 7987 7987
=======================================
+ Hits 49614 49617 +3
Misses 21824 21824
Partials 2050 2050
```
| Flag | Coverage Δ | |
|---|---|---|
| javascript | `53.78% <100.00%> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/superset/pull/23302?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [superset-frontend/src/SqlLab/constants.ts](https://codecov.io/gh/apache/superset/pull/23302?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL1NxbExhYi9jb25zdGFudHMudHM=) | `100.00% <100.00%> (ø)` | |
| [...src/SqlLab/utils/reduxStateToLocalStorageHelper.js](https://codecov.io/gh/apache/superset/pull/23302?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL1NxbExhYi91dGlscy9yZWR1eFN0YXRlVG9Mb2NhbFN0b3JhZ2VIZWxwZXIuanM=) | `100.00% <100.00%> (ø)` | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] ktmud commented on a diff in pull request #23302: fix(sqllab): empty large query results from localStorage
Posted by "ktmud (via GitHub)" <gi...@apache.org>.
ktmud commented on code in PR #23302:
URL: https://github.com/apache/superset/pull/23302#discussion_r1128769453
##########
superset-frontend/src/SqlLab/utils/emptyQueryResults.test.js:
##########
@@ -45,6 +50,38 @@ describe('reduxStateToLocalStorageHelper', () => {
expect(emptiedQuery[id].results).toEqual({});
});
+ it('should empty query.results if query,.results size is greater than LOCALSTORAGE_MAX_QUERY_RESULTS_KB', () => {
+ const reasonableSizeQuery = {
+ ...queries[0],
+ startDttm: Date.now(),
+ results: { data: [{ a: 1 }] },
+ };
+ const largeQuery = {
+ ...queries[1],
+ startDttm: Date.now(),
+ results: {
+ data: [
+ {
+ test: 1123123123123,
+ stringValue:
+ 'Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industrys standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum',
Review Comment:
Can maybe just create a very large array and fill?
```js
new Array(1024*512+1).fill(0).join("")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] justinpark commented on a diff in pull request #23302: fix(sqllab): empty large query results from localStorage
Posted by "justinpark (via GitHub)" <gi...@apache.org>.
justinpark commented on code in PR #23302:
URL: https://github.com/apache/superset/pull/23302#discussion_r1128775578
##########
superset-frontend/src/SqlLab/utils/emptyQueryResults.test.js:
##########
@@ -45,6 +50,38 @@ describe('reduxStateToLocalStorageHelper', () => {
expect(emptiedQuery[id].results).toEqual({});
});
+ it('should empty query.results if query,.results size is greater than LOCALSTORAGE_MAX_QUERY_RESULTS_KB', () => {
+ const reasonableSizeQuery = {
+ ...queries[0],
+ startDttm: Date.now(),
+ results: { data: [{ a: 1 }] },
+ };
+ const largeQuery = {
+ ...queries[1],
+ startDttm: Date.now(),
+ results: {
+ data: [
+ {
+ test: 1123123123123,
+ stringValue:
+ 'Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industrys standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum',
Review Comment:
wow. it's smart.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] john-bodley merged pull request #23302: fix(sqllab): empty large query results from localStorage
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley merged PR #23302:
URL: https://github.com/apache/superset/pull/23302
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org