You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "dcernag (via GitHub)" <gi...@apache.org> on 2024/12/05 18:12:27 UTC
[I] Slack rate-limit when fetching channels [superset]
dcernag opened a new issue, #31309:
URL: https://github.com/apache/superset/issues/31309
### Bug description
When creating or editing a report, superset call is consistently rate-limited by slack when connected to a slack organization with a high number of channels.
Steps to reproduce
1. Enable `ALERT_REPORT_SLACK_V2` feature flag
2. Go to Alerts and Reports
3. Create or edit an existing alert/report and select slack as notification method
Expected behavior:
Dropdown with list of channels is shown and request is retried if rate-limited
Actual behavior:
No dropdown and only one try to get slack channels and exception is shown
I quickly glanced get_channels_with_search and don't immediately see rate-limit handling with retries when error reasong coming from slack is `ratelimited`.
If implemented, it would be amazing to also retry for slack exception reasons: `request_timeout`, `service_unavailable`, and `internal_error`
### Screenshots/recordings
_No response_
### Superset version
master / latest-dev
### Python version
3.10
### Node version
I don't know
### Browser
Chrome
### Additional context
Stack trace:
```
2024-12-05 17:39:03,252:ERROR:superset.reports.api:Error fetching slack channels Failed to list channels: The request to the Slack API failed. (url: https://slack.com/api/conversations.list)
The server responded with: {'ok': False, 'error': 'ratelimited'}
[05/Dec/2024:17:39:03 +0000] "GET /api/v1/report/slack_channels/?q=(exactMatch:!f,searchString:%27%27,types:!(public_channel,private_channel)) HTTP/1.1" 422 190 "https://superset.example.com/report/list/?pageIndex=0&sortColumn=name&sortOrder=desc" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
```
```
FEATURE_FLAGS = {
"ALERT_REPORT_TABS": True,
"ALERT_REPORTS": True,
"CHART_PLUGINS_EXPERIMENTAL": True,
"DASHBOARD_CROSS_FILTERS": True,
"DASHBOARD_RBAC": True,
"DRILL_TO_DETAIL": True,
"DRILL_BY": True,
"ENABLE_TEMPLATE_PROCESSING": True,
"ESTIMATE_QUERY_COST": True,
"EMBEDDED_SUPERSET": True,
"HORIZONTAL_FILTER_BAR": True,
"ALERT_REPORT_SLACK_V2": True,
"TAGGING_SYSTEM": True,
"THUMBNAILS": True,
"THUMBNAILS_SQLA_LISTENERS": True
}
```
### Checklist
- [X] I have searched Superset docs and Slack and didn't find a solution to my problem.
- [X] I have searched the GitHub issue tracker and didn't find a similar bug report.
- [X] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Slack rate-limit when fetching channels [superset]
Posted by "geido (via GitHub)" <gi...@apache.org>.
geido commented on issue #31309:
URL: https://github.com/apache/superset/issues/31309#issuecomment-2528813429
@eschutho I am wondering if you might have some ideas for this problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Slack rate-limit when fetching channels [superset]
Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
dosubot[bot] commented on issue #31309:
URL: https://github.com/apache/superset/issues/31309#issuecomment-2521095941
<!-- Answer -->
To address the issue of Slack API rate-limit errors when fetching channels in Superset, you can implement retry logic using the `backoff` library. This library is already imported in the `slack.py` file, and you can use the `@backoff.on_exception` decorator to handle retries for specific exceptions like `SlackApiError`, `SlackClientError`, and others related to rate limits and transient errors.
Here's an example of how you can implement this:
```python
import backoff
from slack_sdk.errors import SlackApiError, SlackClientError
@backoff.on_exception(
backoff.expo,
(SlackApiError, SlackClientError),
factor=10,
base=2,
max_tries=5,
jitter=backoff.full_jitter,
)
def get_channels_with_search(...) -> list[str]:
# Your existing logic to fetch channels
...
```
This code uses exponential backoff with a factor of 10, a base of 2, and a maximum of 5 retries. It also includes jitter to randomize the wait time between retries, which can help to avoid thundering herd problems. This setup will help in handling rate-limit errors (`ratelimited`) and other transient issues like `request_timeout`, `service_unavailable`, and `internal_error` <sup>[[1]](https://github.com/apache/superset/blob/master/superset/reports/notifications/slack.py)</sup>.
<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Slack rate-limit when fetching channels [superset]
Posted by "Vitor-Avila (via GitHub)" <gi...@apache.org>.
Vitor-Avila commented on issue #31309:
URL: https://github.com/apache/superset/issues/31309#issuecomment-2701750749
This is hopefully going to be solved with this: https://github.com/apache/superset/pull/32510
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Slack rate-limit when fetching channels [superset]
Posted by "Vitor-Avila (via GitHub)" <gi...@apache.org>.
Vitor-Avila commented on issue #31309:
URL: https://github.com/apache/superset/issues/31309#issuecomment-2704363242
Going to be closing this as https://github.com/apache/superset/pull/32510 was merged. Also actively working on caching the slack list to also prevent this issue here https://github.com/apache/superset/pull/32529
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Slack rate-limit when fetching channels [superset]
Posted by "Vitor-Avila (via GitHub)" <gi...@apache.org>.
Vitor-Avila closed issue #31309: Slack rate-limit when fetching channels
URL: https://github.com/apache/superset/issues/31309
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Slack rate-limit when fetching channels [superset]
Posted by "Vitor-Avila (via GitHub)" <gi...@apache.org>.
Vitor-Avila commented on issue #31309:
URL: https://github.com/apache/superset/issues/31309#issuecomment-3637801047
@lariskovski I don't see any version label in those PRs, so I imagine it will be part of `6.0` once it gets approved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Slack rate-limit when fetching channels [superset]
Posted by "lariskovski (via GitHub)" <gi...@apache.org>.
lariskovski commented on issue #31309:
URL: https://github.com/apache/superset/issues/31309#issuecomment-3634729777
@Vitor-Avila in which version are those available? I'm currently having this issue in 4.1.4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org