You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "ganeshvaidee (via GitHub)" <gi...@apache.org> on 2025/11/21 17:27:31 UTC

[I] Celery pods start generating Caching dashboard requests every second without errors or pauses [superset]

ganeshvaidee opened a new issue, #36224:
URL: https://github.com/apache/superset/issues/36224

   ### Bug description
   
   The bug is with version 4.1.1.
   
   For some dashboards, at random times, Celery pods start generating Caching dashboard requests every second without errors or pauses. Here is a sample log
   
   _Caching dashboard: [https://<instance>:443/superset/dashboard/<dashboard](https://%3Cinstance%3E/superset/dashboard/%3Cdashboard) ID>_
   
   This creates a lot of load on a Celery pod, which causes frequent restarts. 
   
   The order of events:
   
   1. Task cache_dashboard_thumbnail <thumbnail id> received
   2. Caching dashboard: [https://<instance>:443/superset/dashboard/<dashboard](https://%3Cinstance%3E/superset/dashboard/%3Cdashboard) ID>/
   3. Task cache_dashboard_thumbnail <thumbnail id> succeeded
   
   And right after, we see
   
   _Task cache_dashboard_thumbnail <new thumbnail ID> received_
   
   ... the cycle continues every second ... 
   
   Sometimes the cycling stops on its own, but it can continue for a long period of time. 
   
   We are looking for some clues on what could be triggering this behavior. 
   
   This happens for published dashboards as well as drafts. We have not set any specific cache timeout setttings.
   
   Is this a known issue? Are there any workarounds or fixes (without a version upgrade) to fix this issue?
   
   ### Screenshots/recordings
   
   _No response_
   
   ### Superset version
   
   4.1.3
   
   ### Python version
   
   3.10
   
   ### Node version
   
   16
   
   ### Browser
   
   Chrome
   
   ### Additional context
   
   _No response_
   
   ### Checklist
   
   - [ ] I have searched Superset docs and Slack and didn't find a solution to my problem.
   - [ ] I have searched the GitHub issue tracker and didn't find a similar bug report.
   - [ ] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] Celery pods start generating Caching dashboard requests every second without errors or pauses [superset]

Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
dosubot[bot] commented on issue #36224:
URL: https://github.com/apache/superset/issues/36224#issuecomment-3564019973

   <!-- Answer -->
   This isn't a widely reported Superset bug, but it's often linked to deployment or configuration issues in distributed setups. The cache_dashboard_thumbnail task is triggered when a dashboard thumbnail is requested and the cache is missing, expired, or in an error state. If the cache never transitions to "UPDATED" (for example, due to a cache backend misconfiguration, Redis key prefix mismatch, or repeated dashboard updates), every request to the thumbnail endpoint will enqueue a new Celery task, causing a storm of requests [reference](https://github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset/dashboards/api.py#L1028-L1327).
   
   Key things to check:
   
   - Make sure your UI cache and Celery results backend use the same Redis key prefix. A mismatch can prevent the cache from being updated, leading to repeated task scheduling [reference](https://github.com/apache/superset/discussions/32595).
   - Only run one instance of Celery Beat in your deployment, and ensure all Superset web and worker nodes share the same metadata database [reference](https://github.com/apache/superset/discussions/32648).
   - If you have the THUMBNAILS_SQLA_LISTENERS feature flag enabled, dashboard updates (including drafts) will trigger thumbnail cache tasks. Frequent updates or event listener misfires can cause repeated scheduling [reference](https://github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset/models/dashboard.py#L352-L475).
   - If you use embedding or async queries, cross-domain setups or guest token issues can also cause repeated or failed cache/task cycles [reference](https://github.com/apache/superset/issues/33561).
   
   Workarounds: align Redis key prefixes, ensure only one Celery Beat, and review your feature flags and cache config. If the problem persists, try disabling THUMBNAILS_SQLA_LISTENERS and see if the storm stops. 
   
   Let me know if you want more details on any of these checks.
   
   
   <!-- Dosu Comment Footer -->
   *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).*
   
   ---
   <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/505dc63d-a02e-462a-acc1-ce4f1371ae16?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/505dc63d-a02e-462a-acc1-ce4f1371ae16?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/505dc63d-a02e-462a-acc1-ce4f1371ae16?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/505dc63d-a02e-462a-acc1-ce4f1371ae16?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/505dc63d-a02e-462a-acc1-ce4f1371ae16?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/505dc63d-a02e-462a-acc1-ce4f1371ae16?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/505dc63d-a02e-462a-acc1-ce4f1371ae16?feedback_type=other)</sup>&nbsp;&nbsp;[![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/a28d3c7e-a9d3-459e-9fb6-3a6f9ff4f357/ask?utm_source=githu
 b)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/apache/superset/issues/36224)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] Celery pods start generating Caching dashboard requests every second without errors or pauses [superset]

Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas closed issue #36224: Celery pods start generating Caching dashboard requests every second without errors or pauses
URL: https://github.com/apache/superset/issues/36224


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] Celery pods start generating Caching dashboard requests every second without errors or pauses [superset]

Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #36224:
URL: https://github.com/apache/superset/issues/36224#issuecomment-3572116650

   Let us know if the bot is right or wrong here... and keep the discussion going with it. We sure haven't seen this one otherwise. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] Celery pods start generating Caching dashboard requests every second without errors or pauses [superset]

Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #36224:
URL: https://github.com/apache/superset/issues/36224#issuecomment-4726466465

   Closing as stale... nothing here to move on as far as I can tell, as we haven't been able to reproduce this on our end. 
   
   If you want this reppened can you share your cache + Celery setup (Redis key prefixes, whether `THUMBNAILS_SQLA_LISTENERS` is on, and that only one Beat is running)? That storm usually points at a cache that never flips to `UPDATED`. This might be better suited as a Discussion if you want to go down that road.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org