You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2024/05/09 19:37:00 UTC
[jira] [Commented] (NIFI-13146) ConsumeSlack processor rate limited
[ https://issues.apache.org/jira/browse/NIFI-13146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845107#comment-17845107 ]
ASF subversion and git services commented on NIFI-13146:
--------------------------------------------------------
Commit 5a3b47353e66d124dd09a1ab4cff75b661911adc in nifi's branch refs/heads/main from krisztina-zsihovszki
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=5a3b47353e ]
NIFI-13146 ConsumeSlack rate limit error mitigation (#8748)
Co-authored-by: Krisztina Zsihovszki <kz...@cloudera.com>
> ConsumeSlack processor rate limited
> -----------------------------------
>
> Key: NIFI-13146
> URL: https://issues.apache.org/jira/browse/NIFI-13146
> Project: Apache NiFi
> Issue Type: Improvement
> Reporter: Zsihovszki Krisztina
> Assignee: Zsihovszki Krisztina
> Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> ConsumeSlack processor was not able to start when it was running in a Slack workplace with thousands of channels. It got stuck in initialization phase and reported rate limit error continuously.
> The processor fetches all available channels (conversation list) during its setup and creates a channel id/name mapping.
> Fetching the conversation list items is executed in batches, 1000 channels/batch. Since the fetch is done continously, after a while Slack API returns rate limit error. (Rate limit settings were at default in Slack.)
> Even if some delay was added after each API call, 30 seconds was not enough to fetch all the channels (since it is in onScheduled, after 30 seconds the initialization is re-attempted)
> As a mitigation for the problem I'd like to add a logic which checks if "Channels" property contains only IDs. In case no channel name is specified, another Slack API call (
> [conversations.info|https://api.slack.com/methods/conversations.info]) can be used to fetch the channel names for the channel IDs and it is not necessary to fetch the whole conversation list.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)