[jira] [Commented] (NIFI-731) If content repo is unable to destroy content as fast as it is generated, nifi performance becomes very erratic

["Mark Payne (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14612081#comment-14612081 ] 

Mark Payne commented on NIFI-731:
---------------------------------

The patch provided provides better performance for me, running Windows 8.1 on both my desktop and laptop, when using magnetic drives. Using an SSD drive, it provides still better performance but the improvement is not nearly as stark as those statistics provided above.

Moreover, it appears that when running in Linux (CentOS) and Mac OSX, this patch actually results in significantly reduce performance.

As a result, I suggest that we not include this patch in this baseline.

A better solution, as noted above, will come in NIFI-744, which is currently slated for version 0.3.0, as it is a much more involved change.

In the meantime, those affected by the issue can make the most out of the current state by adhering to the following recommendations:
* When possible, configure Processors (especially "source processors") to use a Run Duration (in the Settings tab of the configuration) of "25 ms" rather than 0.
* In nifi.properties, change the value of the "nifi.flowfile.repository.checkpoint.interval" property from "2 mins" to "30 secs". This tends to provide very significant performance gains and smoother performance.
* In nifi.properties, change the value of the "nifi.content.repository.archive.enabled" property from "true" to "false" - this will disable archiving of content. though is will improve performance, the content will not be available for view, download, or replay from the Data Provenance UI.

> If content repo is unable to destroy content as fast as it is generated, nifi performance becomes very erratic
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: NIFI-731
>                 URL: https://issues.apache.org/jira/browse/NIFI-731
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Core Framework
>            Reporter: Mark Payne
>            Assignee: Mark Payne
>             Fix For: 0.2.0
>
>         Attachments: 0001-NIFI-731-Updated-admin-guide-to-explain-the-flowfile.patch
>
>
> When the FlowFile Repository marks claims as destructable, it puts the notification on a queue that the content repo pulls from. If the content repo cannot keep up, the queue will fill, resulting in backpressure, that prevents the FlowFile repository from being updated. This, in turn, causes Processors to block, waiting on space to become available. This is by design.
> However, the capacity of this queue is quite large, and the content repo drains the entire queue, then destroys all content claims that are on it. As a result, this act of destroying claims can be quite long, and Processors can block for quite a period of time, leading to very sporadic performance.
> Instead, the content repo should pull from the queue and destroy the claims one at a time or in small batches, instead of draining the entire queue each time. This should result in much less sporadic behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)