[jira] [Commented] (NIFI-1315) Content Repository appears to not apply strict enough backpressure

["Joseph Witt (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-1315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15066650#comment-15066650 ] 

Joseph Witt commented on NIFI-1315:
-----------------------------------

a little more context...

This is on an EC2 cluster (3 nodes) where we've setup a relatively high throughput flow to run in sustained mode.  The problem just occurred again on another box with the same symptoms where it suddenly says no space left on device and starts rolling back transactions and all gets handled nicely.  However, there should be no reason why we run out of space.  I see no evidence of this actually happening as the goals/target for datasize seem to be getting monitored/handled well.

We need a sustained monitoring mechanism to see if space usage ever does fluctuate.  I am presently a little skeptical that we truly did run out of space.  Seems more likely something is occuring on the systems which spuriously tricks NiFi into thinking it is out of space.

> Content Repository appears to not apply strict enough backpressure
> ------------------------------------------------------------------
>
>                 Key: NIFI-1315
>                 URL: https://issues.apache.org/jira/browse/NIFI-1315
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Core Framework
>    Affects Versions: 0.4.0
>            Reporter: Mark Payne
>             Fix For: 0.5.0
>
>
> I was able to setup a dataflow running at a high rate, with CPU load average about double the number of CPUs. With the content repository configured to archive to the point of 80% disk utilization, I still eventually ran out of disk space. NiFi did handle it properly, rolling back sessions and retrying until the disk space was reclaimed from the archive, but it appears that our archive does not apply tight enough constraints for this backpressure, as it should never have gotten to the point of being out of disk space.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)