[jira] [Commented] (NIFI-731) If content repo is unable to destroy content as fast as it is generated, nifi performance becomes very sporadic

["ASF subversion and git services (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14604801#comment-14604801 ] 

ASF subversion and git services commented on NIFI-731:
------------------------------------------------------

Commit bb5128be2246dc48d5200e98572c47ff7cdfb905 in incubator-nifi's branch refs/heads/NIFI-731 from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-nifi.git;h=bb5128b ]

NIFI-731: Updated admin guide to explain the flowfile repo properties more; added new property for now often to 'sync' the flowfile repo; allowed number of partitions in flowfile repo to be changed after repo created. Minor tweaks to content repo and its communication with flowfile repo via ContentClaimManager


> If content repo is unable to destroy content as fast as it is generated, nifi performance becomes very sporadic
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: NIFI-731
>                 URL: https://issues.apache.org/jira/browse/NIFI-731
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Core Framework
>            Reporter: Mark Payne
>            Assignee: Mark Payne
>             Fix For: 0.2.0
>
>
> When the FlowFile Repository marks claims as destructable, it puts the notification on a queue that the content repo pulls from. If the content repo cannot keep up, the queue will fill, resulting in backpressure, that prevents the FlowFile repository from being updated. This, in turn, causes Processors to block, waiting on space to become available. This is by design.
> However, the capacity of this queue is quite large, and the content repo drains the entire queue, then destroys all content claims that are on it. As a result, this act of destroying claims can be quite long, and Processors can block for quite a period of time, leading to very sporadic performance.
> Instead, the content repo should pull from the queue and destroy the claims one at a time or in small batches, instead of draining the entire queue each time. This should result in much less sporadic behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)