You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "Vitor-Avila (via GitHub)" <gi...@apache.org> on 2025/03/06 15:16:57 UTC
[PR] chore: Caching the Slack channels list [superset]
Vitor-Avila opened a new pull request, #32529:
URL: https://github.com/apache/superset/pull/32529
### SUMMARY
The Slack API used via the SDK to retrieve channels has a 20+ requests/minute rate limit, which can be hit pretty fast for large Organizations (there's no filtering in the search, so all channels are parsed with a max of 999 items per page).
Since this request might be executed several times (and in parallel) while reports are getting migrated to V2, this PR implements caching for the list with 1d duration. It also adds a force refresh button the UI dropdown (thanks @eschutho 🙏 ).
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
<!--- Skip this if not applicable -->
### TESTING INSTRUCTIONS
<!--- Required! What steps can be taken to manually verify the changes? -->
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore: Caching the Slack channels list [superset]
Posted by "korbit-ai[bot] (via GitHub)" <gi...@apache.org>.
korbit-ai[bot] commented on code in PR #32529:
URL: https://github.com/apache/superset/pull/32529#discussion_r1983558808
##########
superset/utils/slack.py:
##########
@@ -54,60 +56,82 @@ def get_slack_client() -> WebClient:
return client
+@cache_util.memoized_func(
+ key="slack_conversations_list",
+ cache=cache_manager.cache,
+)
+def get_channels(limit: int, extra_params: dict[str, Any]) -> list[SlackChannelSchema]:
Review Comment:
### Missing Cache Timeout in Slack Channels Cache <sub></sub>
<details>
<summary>Tell me more</summary>
###### What is the issue?
The @cache_util.memoized_func decorator is applied without a cache_timeout parameter, which could lead to stale data being served indefinitely.
###### Why this matters
Without a cache timeout, the Slack channels list will remain cached forever unless manually refreshed, potentially serving outdated channel information to users.
###### Suggested change ∙ *Feature Preview*
Add cache_timeout parameter to the decorator to ensure cache is periodically refreshed:
```python
@cache_util.memoized_func(
key="slack_conversations_list",
cache=cache_manager.cache,
cache_timeout=86400, # 24 hours in seconds
)
```
</details>
<sub>
[](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/a1f382c1-763c-4d97-bc49-9008a9db0e5b?suggestedFixEnabled=true)
💬 Looking for more details? Reply to this comment to chat with Korbit.
</sub>
<!--- korbi internal id:2a66ce08-db70-49c9-b4eb-767220d3c8b6 -->
##########
superset-frontend/src/features/alerts/components/NotificationMethod.tsx:
##########
@@ -248,61 +254,84 @@ export const NotificationMethod: FunctionComponent<NotificationMethodProps> = ({
searchString = '',
types = [],
exactMatch = false,
+ force = false,
}: {
searchString?: string | undefined;
types?: string[];
exactMatch?: boolean | undefined;
+ force?: boolean | undefined;
} = {}): Promise<JsonResponse> => {
- const queryString = rison.encode({ searchString, types, exactMatch });
+ const queryString = rison.encode({
+ searchString,
+ types,
+ exactMatch,
+ force,
+ });
const endpoint = `/api/v1/report/slack_channels/?q=${queryString}`;
return SupersetClient.get({ endpoint });
};
+ const updateSlackOptions = async ({
+ force,
+ }: {
+ force?: boolean | undefined;
+ } = {}) => {
+ if (force) {
+ addInfoToast(
+ t('Fetching Slack channels. This operation may take a while.'),
+ );
+ }
+ fetchSlackChannels({ types: ['public_channel', 'private_channel'], force })
+ .then(({ json }) => {
+ const { result } = json;
+ const options: SlackOptionsType = mapChannelsToOptions(result);
+
+ setSlackOptions(options);
+
+ if (isFeatureEnabled(FeatureFlag.AlertReportSlackV2)) {
+ // for edit mode, map existing ids to names for display if slack v2
+ // or names to ids if slack v1
+ const [publicOptions, privateOptions] = options;
+ if (
+ method &&
+ [
+ NotificationMethodOption.SlackV2,
+ NotificationMethodOption.Slack,
+ ].includes(method)
+ ) {
+ setSlackRecipients(
+ mapSlackValues({
+ method,
+ recipientValue,
+ slackOptions: [
+ ...publicOptions.options,
+ ...privateOptions.options,
+ ],
+ }),
+ );
+ }
+ }
+ })
Review Comment:
### Silent error handling in Slack API call <sub></sub>
<details>
<summary>Tell me more</summary>
###### What is the issue?
The catch block silently swallows the error details when the Slack API call fails, only setting useSlackV1 to true.
###### Why this matters
Without logging or surfacing the actual error details, debugging Slack API integration issues becomes difficult and time-consuming.
###### Suggested change ∙ *Feature Preview*
Add error logging and surface the error details to users:
```typescript
.catch(error => {
// Fallback to slack v1 if slack v2 is not compatible
setUseSlackV1(true);
addInfoToast(t('Failed to fetch Slack channels. Falling back to legacy Slack integration.'));
console.error('Slack channels fetch error:', error);
})
```
</details>
<sub>
[](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/fbfacb37-51b7-48ea-a7d3-4d02b8b57d9e?suggestedFixEnabled=true)
💬 Looking for more details? Reply to this comment to chat with Korbit.
</sub>
<!--- korbi internal id:90054be6-621b-4b1b-8945-1b5b94da7b91 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore: Caching the Slack channels list [superset]
Posted by "Vitor-Avila (via GitHub)" <gi...@apache.org>.
Vitor-Avila commented on code in PR #32529:
URL: https://github.com/apache/superset/pull/32529#discussion_r1983814388
##########
superset/utils/slack.py:
##########
@@ -54,60 +56,82 @@ def get_slack_client() -> WebClient:
return client
+@cache_util.memoized_func(
+ key="slack_conversations_list",
+ cache=cache_manager.cache,
+)
+def get_channels(limit: int, extra_params: dict[str, Any]) -> list[SlackChannelSchema]:
+ """
+ Retrieves a list of all conversations accessible by the bot
+ from the Slack API, and caches results (to avoid rate limits).
+
+ The Slack API does not provide search so to apply a search use
+ get_channels_with_search instead.
+ """
+ client = get_slack_client()
+ channel_schema = SlackChannelSchema()
+ channels: list[SlackChannelSchema] = []
+ cursor = None
+
+ while True:
+ response = client.conversations_list(
+ limit=limit, cursor=cursor, exclude_archived=True, **extra_params
+ )
+ channels.extend(
+ channel_schema.load(channel) for channel in response.data["channels"]
+ )
+ cursor = response.data.get("response_metadata", {}).get("next_cursor")
+ if not cursor:
+ break
+
+ return channels
+
+
def get_channels_with_search(
search_string: str = "",
limit: int = 999,
types: Optional[list[SlackChannelTypes]] = None,
exact_match: bool = False,
+ force: bool = False,
) -> list[SlackChannelSchema]:
"""
The slack api is paginated but does not include search, so we need to fetch
all channels and filter them ourselves
This will search by slack name or id
"""
-
+ extra_params = {}
+ extra_params["types"] = ",".join(types) if types else None
try:
- client = get_slack_client()
- channel_schema = SlackChannelSchema()
- channels: list[SlackChannelSchema] = []
- cursor = None
- extra_params = {}
- extra_params["types"] = ",".join(types) if types else None
-
- while True:
- response = client.conversations_list(
- limit=limit, cursor=cursor, exclude_archived=True, **extra_params
- )
- channels.extend(
- channel_schema.load(channel) for channel in response.data["channels"]
- )
- cursor = response.data.get("response_metadata", {}).get("next_cursor")
- if not cursor:
- break
-
- # The search string can be multiple channels separated by commas
- if search_string:
- search_array = recipients_string_to_list(search_string)
- channels = [
- channel
- for channel in channels
- if any(
- (
- search.lower() == channel["name"].lower()
- or search.lower() == channel["id"].lower()
- if exact_match
- else (
- search.lower() in channel["name"].lower()
- or search.lower() in channel["id"].lower()
- )
- )
- for search in search_array
- )
- ]
- return channels
+ channels = get_channels(
+ limit=limit,
+ extra_params=extra_params,
+ force=force,
+ cache_timeout=86400,
Review Comment:
Hardcoded here for a day, which I think should be generally good. I could make this a dynamic config in a follow up if we think it's better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore: Caching the Slack channels list [superset]
Posted by "Vitor-Avila (via GitHub)" <gi...@apache.org>.
Vitor-Avila merged PR #32529:
URL: https://github.com/apache/superset/pull/32529
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore: Caching the Slack channels list [superset]
Posted by "Vitor-Avila (via GitHub)" <gi...@apache.org>.
Vitor-Avila commented on code in PR #32529:
URL: https://github.com/apache/superset/pull/32529#discussion_r1983703958
##########
superset/utils/slack.py:
##########
@@ -54,60 +56,82 @@ def get_slack_client() -> WebClient:
return client
+@cache_util.memoized_func(
+ key="slack_conversations_list",
+ cache=cache_manager.cache,
+)
+def get_channels(limit: int, extra_params: dict[str, Any]) -> list[SlackChannelSchema]:
Review Comment:
This is already hardcoded where `get_channels` is called. Trying to hardcode it at the decorator level would cause an error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore: Caching the Slack channels list [superset]
Posted by "korbit-ai[bot] (via GitHub)" <gi...@apache.org>.
korbit-ai[bot] commented on code in PR #32529:
URL: https://github.com/apache/superset/pull/32529#discussion_r1983704521
##########
superset/utils/slack.py:
##########
@@ -54,60 +56,82 @@ def get_slack_client() -> WebClient:
return client
+@cache_util.memoized_func(
+ key="slack_conversations_list",
+ cache=cache_manager.cache,
+)
+def get_channels(limit: int, extra_params: dict[str, Any]) -> list[SlackChannelSchema]:
Review Comment:
Thanks for your response, Vitor-Avila. If the cache_timeout parameter is being set where `get_channels` is called, you can disregard my previous comment about hardcoding it in the decorator. However, it's essential to make sure the cache is being appropriately invalidated or refreshed to avoid serving redundant information. If possible, could you please share the relevant code section for a more comprehensive review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org