[jira] [Resolved] (NIFI-6905) GetTwitter processor, configured to run on primary node only, initializes connection to Twitter API from every NiFi cluster node, even on non-primary nodes

["Mark Payne (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NIFI-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Payne resolved NIFI-6905.
------------------------------
    Fix Version/s: 1.11.0
       Resolution: Fixed

> GetTwitter processor, configured to run on primary node only, initializes connection to Twitter API from every NiFi cluster node, even on non-primary nodes
> -----------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: NIFI-6905
>                 URL: https://issues.apache.org/jira/browse/NIFI-6905
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Extensions
>    Affects Versions: 1.0.0
>            Reporter: Kourge
>            Assignee: Kourge
>            Priority: Major
>              Labels: getTwitter
>             Fix For: 1.11.0
>
>          Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> I have a *GetTwitter* processor running on a 3-nodes NiFi cluster and configured to be executed on the primary node only.
> The symptom is that there is a too high frequency of HTTP 420 ("Enhance Your Calm") exceptions on GetTwitter processor start.
> I made the following tests:
>  * With only 1 NiFi node. I was able to start/stop GetTwitter processor 10 times in a raw without any errors.
>  * With 2 NiFi nodes running, HTTP 420 errors occurred after a few start/stop (sometimes even after a single start).
> After an analysis of the source code and knowing https://issues.apache.org/jira/browse/NIFI-2592 I came to the conclusion that the GetTwitter processor is initializing the connection to Twitter API on each node of the cluster, even to non-primary nodes.
> The `*onScheduled()*` method is run on every node (see: NIFI-2592) making connections to Twitter with `*client.connect()*`. Then the `*onTrigger()*` method consumes the tweets normally from the primary node.
> Issue is that having more that one node initializing connections make Twitter API raise HTTP 420 errors.
> {code:java}
> ERROR
> org.apache.nifi.processors.twitter.GetTwitter
> GetTwitter[id=XYZ] Received error HTTP_ERROR: HTTP/1.1 420 Enhance Your Calm. Will attempt to reconnect
> {code}
> +*Proposed solutions:*+
>  # Change the behavior of `*onScheduled()*` method to run only on primary node (as proposed in NIFI-2592)
>  # Update GetTwitter processor implementation to not call `*client.connect()*` anymore from the `*onScheduled()*` method but only when *PrimaryNodeState* changes to *ELECTED_PRIMARY_NODE* (And when *PrimaryNodeState* changes to *PRIMARY_NODE_REVOKED*: perform a `*client.stop()*`)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)