You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Boris Tyukin (JIRA)" <ji...@apache.org> on 2018/03/06 19:55:00 UTC
[jira] [Commented] (NIFI-90) Replace explicit penalization with
automatic penalization/back-off
[ https://issues.apache.org/jira/browse/NIFI-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16388431#comment-16388431 ]
Boris Tyukin commented on NIFI-90:
----------------------------------
I am surprised this Jira has not got any traction after 3 years...Having used Apache Airflow for a while, I am looking to retry capabilities in NiFi and it comes down to "building your own" flows, that would handle retries in a loop and then sleeping for some time. The best solution I found, suggested by [Alessio Palma|https://community.hortonworks.com/users/16011/ozw1z5rd.html] [https://community.hortonworks.com/questions/56167/is-there-wait-processor-in-nifi.html] but still would be nice to have re-try capabilities like Airflow folks. You can specify global re-try behavior for a flow or override per task/processor. This helps a lot to deal with intermittent issues, like losing network connection or source database system, being down for maintenance.
> Replace explicit penalization with automatic penalization/back-off
> ------------------------------------------------------------------
>
> Key: NIFI-90
> URL: https://issues.apache.org/jira/browse/NIFI-90
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Reporter: Joseph Witt
> Priority: Minor
>
> Rather than having users configure explicit penalization periods and requiring developers to implement it in their processors we can automate this. Perhaps keep a LinkedHashMap<Connection ID, Counter> of size 5 or so in the FlowFileRecord construct. When a FlowFile is routed to a Connection, the counter is incremented. If the counter exceeds 3 visits to the same connection, the FlowFile will be automatically penalized. This protects us "5 hops out" so that if we have something like DistributeLoad -> PostHTTP with failure looping back to DistributeLoad, we will still penalize when appropriate.
> In addition, we will remove the configuration option from the UI, setting the penalization period to some default such as 5 seconds.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)