You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2022/07/18 09:00:30 UTC
[GitHub] [superset] konradwudkowski opened a new issue, #20741: High memory usage when returning results
konradwudkowski opened a new issue, #20741:
URL: https://github.com/apache/superset/issues/20741
When superset returns results it converts nested fields to json with [this line of code](https://github.com/apache/superset/blob/f3b289d3c333fe2351e9fbac6fa85b875cb1897c/superset/result_set.py#L65-L67)
This results in **excessive** amount of memory used and kills celery worker. The problem manifests itself for large enough number of records. I noticed it on Presto with big highly nested records. I suspect it may show on other databases.
I realize that more records means more memory but it my tests selecting 200k records (this value will vary based on data itself) resulted in megabytes of memory used until SupersetResultSet is [created](https://github.com/apache/superset/blob/f3b289d3c333fe2351e9fbac6fa85b875cb1897c/superset/sql_lab.py#L316) and this explodes to gigabytes and was failing when celery worker had 10gb of memory allocated.
#### How to reproduce the bug
I encountered the problem on Presto with big highly nested records.
1. Create a dataset with many records and nested fields (so that this line is executed https://github.com/apache/superset/blob/f3b289d3c333fe2351e9fbac6fa85b875cb1897c/superset/result_set.py#L142)
2. For high enough number of records selected superset will OOM (in our case celery worker so query fails but superset itself lives on)
Error Stack trace
```
celeryworker_1 | [2022-07-12 12:42:15,512: INFO/ForkPoolWorker-1] Query 5: Polling the cursor for progress
celeryworker_1 | [2022-07-12 12:42:40,194: ERROR/MainProcess] Process 'ForkPoolWorker-1' pid:12 exited with 'signal 9 (SIGKILL)'
celeryworker_1 | [2022-07-12 12:42:41,037: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 9 (SIGKILL) Job: 3.')
celeryworker_1 | Traceback (most recent call last):
celeryworker_1 | File "/opt/superset/billiard/pool.py", line 1265, in mark_as_worker_lost
celeryworker_1 | raise WorkerLostError(
celeryworker_1 | billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL) Job: 3.
```
### Expected results
I expect that a result set that takes several megabytes of memory before processing will not blow.
### Actual results
OOM on celery worker
### Environment
(please complete the following information):
- browser type and version: any
- superset version: `1.5.1`
- python version: `3.8.12`
### Checklist
Make sure to follow these steps before submitting your issue - thank you!
- [x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
- [x] I have reproduced the issue with at least the latest released version of superset.
- [x] I have checked the issue tracker for the same issue and I haven't found one similar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] High memory usage when returning results [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas closed issue #20741: High memory usage when returning results
URL: https://github.com/apache/superset/issues/20741
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] High memory usage when returning results [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas closed issue #20741: High memory usage when returning results
URL: https://github.com/apache/superset/issues/20741
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] High memory usage when returning results [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #20741:
URL: https://github.com/apache/superset/issues/20741#issuecomment-2218678713
Closing as stale. Holler if this needs a revisit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] anandnalya commented on issue #20741: High memory usage when returning results
Posted by "anandnalya (via GitHub)" <gi...@apache.org>.
anandnalya commented on issue #20741:
URL: https://github.com/apache/superset/issues/20741#issuecomment-1473115780
I ran into the same issues in Superset 2.0.1 when the results contained a cell with an array of ~30,000 integers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] High memory usage when returning results [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #20741:
URL: https://github.com/apache/superset/issues/20741#issuecomment-1962024351
Is anyone still experiencing this in Superset 3.x? We no longer support earlier versions, so I would normally just close this, but I want to take this seriously if it's still a problem today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org