[jira] [Commented] (NIFI-3897) Queue with flow files thinks it is empty

["Joseph Percivall (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-3897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16061573#comment-16061573 ] 

Joseph Percivall commented on NIFI-3897:
----------------------------------------

I believe I hit this state. I debugged a bit with [~markap14] and I'll let him comment on what he's going to do but we believe it to be related to un-caught throwables (ie. out of memory errors) but not 100% sure it only arises under those conditions. Below I'll outline a summary of what the problem was, what I tried and what fixed it:


NiFi 1.3.0 standalone instance

Apparent problem:
*     An RPG is routed to a funnel and FlowFiles are not being processed by the funnel from that queue. The RPG continues to grab more FlowFiles


Other symptoms: 
*     The funnel will still process flowfiles coming from another connection
*     There are 2 swap files on disk for that queue (logs for the creation of them have aged off)
*     There is a distinct cliff of flowfiles getting processed and then stopping
*     There was a swap for that queue a half second prior to the cliff but is swapped back in a couple seconds later
*     An out of memory error log is present in the bootstrap logs for a couple days prior to the cliff
    
Actions Tried:
*     List queue says there are 0 flowfiles in the queue
*     Moving the connection to another component. Then moving it back. Also tried starting the other component. Neither worked.
*     Setting the expiration had no effect
*     Back-pressure works as expected.
*     Sent in a large group of flowfiles (larger than the swap size). Once it hit the next swap size it flipped and started processing those. It only processed down to 2 swap files worth and then got into the same state.

Remedy:
*     Restarted NiFi and it successfully processed them


> Queue with flow files thinks it is empty
> ----------------------------------------
>
>                 Key: NIFI-3897
>                 URL: https://issues.apache.org/jira/browse/NIFI-3897
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Core Framework
>    Affects Versions: 1.2.0
>            Reporter: Joseph Gresock
>         Attachments: queue-has-no-flowfiles.png
>
>
> A particular queue is listed as having 80,000 flow files, but NiFi thinks there are none to list.  I was originally inclined to believe that the queue is indeed empty, and that there is somehow a pointer to 80,000 flow files leftover on the queue.
> Unfortunately, the logs have rolled over so I can't see what error or warning messages may have happened at the time, and I can't reproduce it reliably.
> Update: I happened to reboot my cluster, and suddenly the 80,000 flow files went through the flow.  This makes the problem considerably worse, since it appears the flow files were "invisible" until the reboot.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)