You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Elli Schwarz <el...@yahoo.com> on 2015/11/06 17:25:49 UTC
Queue wasn't emptying
I had a queue that built up overnight with several thousand flowfiles. The queue was pointing to a RouteOnAttribute processor, which was running. For some reason, the RouteOnAttribute processor wasn't emptying the queue so the queue just built up. Stopping and starting the processor didn't help. However, simply restarting Nifi got the flow moving again. I can't figure out why the queue was stuck.
What could cause a queue to build up like that? How would restarting Nifi get it going again? There weren't any errors in the log. I did have backpressure on the relationship to the processor after the RouteOnAttribute set to 1, but that queue was empty. There was no backpressure on the queue heading in to the RouteOnAttribute processor. The RouteOnAttribute processor has a Penalty Duration and a Yield of 1 sec. The relationships all used the PriorityAttributeAnalyzer as the prioritizer.
The interesting thing is I had the exact same problem on two different nifis (one was version 0.2.0 and one 0.3.0). The same type of flow files were going from one to the other using site-to-site (I was doing this for load balancing). On the other Nifi, instead of restarting, I added another RouteOnAttribute processor and rerouted the stuck queue to the new one instead, and that also got things moving again. (I have a backpressure since the processor after the RouteOnAttribute is a ControlRate processor, and I only want to allow 1 flowfile through per second so as not to overwhelm the system downstream. The backpressure is to assist with load balancing, so if one pathway fills up, I have the other pathway which uses the site-to-site to route the flow to the other nifi).
Is there some combination of backpressure, penalty or yield along with prioritization that could cause a kind of deadlock-like situation? Any ideas as to how I can prevent this from occurring?
Thanks!-Elli
Re: Queue wasn't emptying
Posted by Joe Witt <jo...@gmail.com>.
Elli,
What you may be seeing could be a different thing. In 0.4.0 we've
improved who large queue handling works significantly.
Thanks
Joe
On Thu, Nov 12, 2015 at 8:56 AM, Elli Schwarz <el...@yahoo.com> wrote:
> I had the issue with 0.2.0 and 0.3.0, and I did not build from source. It
> happens on rare occasions, and I can't figure out what causes it. I do have
> backpressure but no expiration on the route that RouteOnAttribute feeds. I
> haven't had a chance to try to dump.
>
> It seems to happen after a long period of time of the queue being full and
> backpressure being applied due to a large amount of flowfiles being added
> constantly.
>
>
>
> On Thursday, November 12, 2015 7:42 AM, Mark Payne <ma...@hotmail.com>
> wrote:
>
>
>
> Chris / Elli,
>
> What version of NiFi are you running? I am wondering if you may have been
> bitten by
> this bug [1]. This should affect only the 'master' branch, not any formally
> released
> versions of NiFi. If you built the application from source code, rather than
> using an
> already-released version, this may well be what is happening.
>
> Thanks
> -Mark
>
>
> [1] https://issues.apache.org/jira/browse/NIFI-1155
>
>
> On Nov 12, 2015, at 2:32 AM, Christopher Hamm <em...@christopherhamm.com>
> wrote:
>
> where does it dump? I cant get mine to expire fast enough. It is dev test I
> want to blow away but cant delete.
>
> On Fri, Nov 6, 2015 at 11:33 AM, Joe Witt <jo...@gmail.com> wrote:
>
> Elli,
>
> Can you share your flow configuration that we could possibly use to
> replicate this? Perhaps turn this into a JIRA and attach that.
>
> Also anytime you see something that appears 'stuck' please try to get
> a stack dump (bin/nifi.sh dump) As if there is truly a stuck thread
> we'll see it there and what it is blocked on.
>
> Thanks
> Joe
>
> On Fri, Nov 6, 2015 at 11:25 AM, Elli Schwarz <el...@yahoo.com>
> wrote:
>> I had a queue that built up overnight with several thousand flowfiles. The
>> queue was pointing to a RouteOnAttribute processor, which was running. For
>> some reason, the RouteOnAttribute processor wasn't emptying the queue so
>> the
>> queue just built up. Stopping and starting the processor didn't help.
>> However, simply restarting Nifi got the flow moving again. I can't figure
>> out why the queue was stuck.
>>
>> What could cause a queue to build up like that? How would restarting Nifi
>> get it going again? There weren't any errors in the log. I did have
>> backpressure on the relationship to the processor after the
>> RouteOnAttribute
>> set to 1, but that queue was empty. There was no backpressure on the queue
>> heading in to the RouteOnAttribute processor. The RouteOnAttribute
>> processor
>> has a Penalty Duration and a Yield of 1 sec. The relationships all used
>> the
>> PriorityAttributeAnalyzer as the prioritizer.
>>
>> The interesting thing is I had the exact same problem on two different
>> nifis
>> (one was version 0.2.0 and one 0.3.0). The same type of flow files were
>> going from one to the other using site-to-site (I was doing this for load
>> balancing). On the other Nifi, instead of restarting, I added another
>> RouteOnAttribute processor and rerouted the stuck queue to the new one
>> instead, and that also got things moving again. (I have a backpressure
>> since
>> the processor after the RouteOnAttribute is a ControlRate processor, and I
>> only want to allow 1 flowfile through per second so as not to overwhelm
>> the
>> system downstream. The backpressure is to assist with load balancing, so
>> if
>> one pathway fills up, I have the other pathway which uses the site-to-site
>> to route the flow to the other nifi).
>>
>> Is there some combination of backpressure, penalty or yield along with
>> prioritization that could cause a kind of deadlock-like situation? Any
>> ideas
>> as to how I can prevent this from occurring?
>>
>> Thanks!
>> -Elli
>>
>
>
>
>
> --
> Sincerely,
> Chris Hamm
> (E) cehamm01@gmail.com
> (Twitter) http://twitter.com/webhamm
> (Linkedin) http://www.linkedin.com/in/chrishamm
>
>
>
>
Re: Queue wasn't emptying
Posted by Elli Schwarz <el...@yahoo.com>.
I had the issue with 0.2.0 and 0.3.0, and I did not build from source. It happens on rare occasions, and I can't figure out what causes it. I do have backpressure but no expiration on the route that RouteOnAttribute feeds. I haven't had a chance to try to dump.
It seems to happen after a long period of time of the queue being full and backpressure being applied due to a large amount of flowfiles being added constantly.
On Thursday, November 12, 2015 7:42 AM, Mark Payne <ma...@hotmail.com> wrote:
Chris / Elli,
What version of NiFi are you running? I am wondering if you may have been bitten bythis bug [1]. This should affect only the 'master' branch, not any formally releasedversions of NiFi. If you built the application from source code, rather than using analready-released version, this may well be what is happening.
Thanks-Mark
[1] https://issues.apache.org/jira/browse/NIFI-1155
On Nov 12, 2015, at 2:32 AM, Christopher Hamm <em...@christopherhamm.com> wrote:
where does it dump? I cant get mine to expire fast enough. It is dev test I want to blow away but cant delete.
On Fri, Nov 6, 2015 at 11:33 AM, Joe Witt <jo...@gmail.com> wrote:
Elli,
Can you share your flow configuration that we could possibly use to
replicate this? Perhaps turn this into a JIRA and attach that.
Also anytime you see something that appears 'stuck' please try to get
a stack dump (bin/nifi.sh dump) As if there is truly a stuck thread
we'll see it there and what it is blocked on.
Thanks
Joe
On Fri, Nov 6, 2015 at 11:25 AM, Elli Schwarz <el...@yahoo.com> wrote:
> I had a queue that built up overnight with several thousand flowfiles. The
> queue was pointing to a RouteOnAttribute processor, which was running. For
> some reason, the RouteOnAttribute processor wasn't emptying the queue so the
> queue just built up. Stopping and starting the processor didn't help.
> However, simply restarting Nifi got the flow moving again. I can't figure
> out why the queue was stuck.
>
> What could cause a queue to build up like that? How would restarting Nifi
> get it going again? There weren't any errors in the log. I did have
> backpressure on the relationship to the processor after the RouteOnAttribute
> set to 1, but that queue was empty. There was no backpressure on the queue
> heading in to the RouteOnAttribute processor. The RouteOnAttribute processor
> has a Penalty Duration and a Yield of 1 sec. The relationships all used the
> PriorityAttributeAnalyzer as the prioritizer.
>
> The interesting thing is I had the exact same problem on two different nifis
> (one was version 0.2.0 and one 0.3.0). The same type of flow files were
> going from one to the other using site-to-site (I was doing this for load
> balancing). On the other Nifi, instead of restarting, I added another
> RouteOnAttribute processor and rerouted the stuck queue to the new one
> instead, and that also got things moving again. (I have a backpressure since
> the processor after the RouteOnAttribute is a ControlRate processor, and I
> only want to allow 1 flowfile through per second so as not to overwhelm the
> system downstream. The backpressure is to assist with load balancing, so if
> one pathway fills up, I have the other pathway which uses the site-to-site
> to route the flow to the other nifi).
>
> Is there some combination of backpressure, penalty or yield along with
> prioritization that could cause a kind of deadlock-like situation? Any ideas
> as to how I can prevent this from occurring?
>
> Thanks!
> -Elli
>
--
Sincerely,
Chris Hamm
(E) cehamm01@gmail.com
(Twitter) http://twitter.com/webhamm
(Linkedin) http://www.linkedin.com/in/chrishamm
Re: Queue wasn't emptying
Posted by Christopher Hamm <ce...@gmail.com>.
yes and thank you
On Nov 12, 2015 8:30 AM, "Mark Payne" <ma...@hotmail.com> wrote:
> Chris,
>
> If this is a test system (which I believe you said it is) and you're okay
> losing all data in the system, you can shutdown
> and delete your content and flowfile repositories (by default they are
> located at
> ./flowfile_repository and ./content_repository) and then restart.
>
> Thanks
> -Mark
>
>
> On Nov 12, 2015, at 8:25 AM, Christopher Hamm <ce...@gmail.com> wrote:
>
> that isn't clearing them out really. it gradually decreases but doesn't
> clear out 500k items.
> On Nov 12, 2015 8:22 AM, "Juan Sequeiros" <ju...@onyxpoint.com>
> wrote:
>
>> To answer Christopher,
>>
>> You can clear out flowfiles on a queue by setting expiration on
>> relationship line.
>>
>> On Thu, Nov 12, 2015 at 7:53 AM, Christopher Hamm <ce...@gmail.com>
>> wrote:
>>
>>> I will look, but how do I just kill it with fire. What can I delete?
>>> On Nov 12, 2015 7:42 AM, "Mark Payne" <ma...@hotmail.com> wrote:
>>>
>>>> Chris / Elli,
>>>>
>>>> What version of NiFi are you running? I am wondering if you may have
>>>> been bitten by
>>>> this bug [1]. This should affect only the 'master' branch, not any
>>>> formally released
>>>> versions of NiFi. If you built the application from source code, rather
>>>> than using an
>>>> already-released version, this may well be what is happening.
>>>>
>>>> Thanks
>>>> -Mark
>>>>
>>>>
>>>> [1] https://issues.apache.org/jira/browse/NIFI-1155
>>>>
>>>>
>>>> On Nov 12, 2015, at 2:32 AM, Christopher Hamm <
>>>> email@christopherhamm.com> wrote:
>>>>
>>>> where does it dump? I cant get mine to expire fast enough. It is dev
>>>> test I want to blow away but cant delete.
>>>>
>>>> On Fri, Nov 6, 2015 at 11:33 AM, Joe Witt <jo...@gmail.com> wrote:
>>>>
>>>>> Elli,
>>>>>
>>>>> Can you share your flow configuration that we could possibly use to
>>>>> replicate this? Perhaps turn this into a JIRA and attach that.
>>>>>
>>>>> Also anytime you see something that appears 'stuck' please try to get
>>>>> a stack dump (bin/nifi.sh dump) As if there is truly a stuck thread
>>>>> we'll see it there and what it is blocked on.
>>>>>
>>>>> Thanks
>>>>> Joe
>>>>>
>>>>> On Fri, Nov 6, 2015 at 11:25 AM, Elli Schwarz <
>>>>> eliezer_schwarz@yahoo.com> wrote:
>>>>> > I had a queue that built up overnight with several thousand
>>>>> flowfiles. The
>>>>> > queue was pointing to a RouteOnAttribute processor, which was
>>>>> running. For
>>>>> > some reason, the RouteOnAttribute processor wasn't emptying the
>>>>> queue so the
>>>>> > queue just built up. Stopping and starting the processor didn't help.
>>>>> > However, simply restarting Nifi got the flow moving again. I can't
>>>>> figure
>>>>> > out why the queue was stuck.
>>>>> >
>>>>> > What could cause a queue to build up like that? How would restarting
>>>>> Nifi
>>>>> > get it going again? There weren't any errors in the log. I did have
>>>>> > backpressure on the relationship to the processor after the
>>>>> RouteOnAttribute
>>>>> > set to 1, but that queue was empty. There was no backpressure on the
>>>>> queue
>>>>> > heading in to the RouteOnAttribute processor. The RouteOnAttribute
>>>>> processor
>>>>> > has a Penalty Duration and a Yield of 1 sec. The relationships all
>>>>> used the
>>>>> > PriorityAttributeAnalyzer as the prioritizer.
>>>>> >
>>>>> > The interesting thing is I had the exact same problem on two
>>>>> different nifis
>>>>> > (one was version 0.2.0 and one 0.3.0). The same type of flow files
>>>>> were
>>>>> > going from one to the other using site-to-site (I was doing this for
>>>>> load
>>>>> > balancing). On the other Nifi, instead of restarting, I added another
>>>>> > RouteOnAttribute processor and rerouted the stuck queue to the new
>>>>> one
>>>>> > instead, and that also got things moving again. (I have a
>>>>> backpressure since
>>>>> > the processor after the RouteOnAttribute is a ControlRate processor,
>>>>> and I
>>>>> > only want to allow 1 flowfile through per second so as not to
>>>>> overwhelm the
>>>>> > system downstream. The backpressure is to assist with load
>>>>> balancing, so if
>>>>> > one pathway fills up, I have the other pathway which uses the
>>>>> site-to-site
>>>>> > to route the flow to the other nifi).
>>>>> >
>>>>> > Is there some combination of backpressure, penalty or yield along
>>>>> with
>>>>> > prioritization that could cause a kind of deadlock-like situation?
>>>>> Any ideas
>>>>> > as to how I can prevent this from occurring?
>>>>> >
>>>>> > Thanks!
>>>>> > -Elli
>>>>> >
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Sincerely,
>>>> Chris Hamm
>>>> (E) cehamm01@gmail.com
>>>> (Twitter) http://twitter.com/webhamm
>>>> (Linkedin) http://www.linkedin.com/in/chrishamm
>>>>
>>>>
>>>>
>>
>
Re: Queue wasn't emptying
Posted by Mark Payne <ma...@hotmail.com>.
Chris,
If this is a test system (which I believe you said it is) and you're okay losing all data in the system, you can shutdown
and delete your content and flowfile repositories (by default they are located at
./flowfile_repository and ./content_repository) and then restart.
Thanks
-Mark
> On Nov 12, 2015, at 8:25 AM, Christopher Hamm <ce...@gmail.com> wrote:
>
> that isn't clearing them out really. it gradually decreases but doesn't clear out 500k items.
>
> On Nov 12, 2015 8:22 AM, "Juan Sequeiros" <juan.sequeiros@onyxpoint.com <ma...@onyxpoint.com>> wrote:
> To answer Christopher,
> You can clear out flowfiles on a queue by setting expiration on relationship line.
>
>
> On Thu, Nov 12, 2015 at 7:53 AM, Christopher Hamm <cehamm01@gmail.com <ma...@gmail.com>> wrote:
> I will look, but how do I just kill it with fire. What can I delete?
>
> On Nov 12, 2015 7:42 AM, "Mark Payne" <markap14@hotmail.com <ma...@hotmail.com>> wrote:
> Chris / Elli,
>
> What version of NiFi are you running? I am wondering if you may have been bitten by
> this bug [1]. This should affect only the 'master' branch, not any formally released
> versions of NiFi. If you built the application from source code, rather than using an
> already-released version, this may well be what is happening.
>
> Thanks
> -Mark
>
>
> [1] https://issues.apache.org/jira/browse/NIFI-1155 <https://issues.apache.org/jira/browse/NIFI-1155>
>
>
>> On Nov 12, 2015, at 2:32 AM, Christopher Hamm <email@christopherhamm.com <ma...@christopherhamm.com>> wrote:
>>
>> where does it dump? I cant get mine to expire fast enough. It is dev test I want to blow away but cant delete.
>>
>> On Fri, Nov 6, 2015 at 11:33 AM, Joe Witt <joe.witt@gmail.com <ma...@gmail.com>> wrote:
>> Elli,
>>
>> Can you share your flow configuration that we could possibly use to
>> replicate this? Perhaps turn this into a JIRA and attach that.
>>
>> Also anytime you see something that appears 'stuck' please try to get
>> a stack dump (bin/nifi.sh dump) As if there is truly a stuck thread
>> we'll see it there and what it is blocked on.
>>
>> Thanks
>> Joe
>>
>> On Fri, Nov 6, 2015 at 11:25 AM, Elli Schwarz <eliezer_schwarz@yahoo.com <ma...@yahoo.com>> wrote:
>> > I had a queue that built up overnight with several thousand flowfiles. The
>> > queue was pointing to a RouteOnAttribute processor, which was running. For
>> > some reason, the RouteOnAttribute processor wasn't emptying the queue so the
>> > queue just built up. Stopping and starting the processor didn't help.
>> > However, simply restarting Nifi got the flow moving again. I can't figure
>> > out why the queue was stuck.
>> >
>> > What could cause a queue to build up like that? How would restarting Nifi
>> > get it going again? There weren't any errors in the log. I did have
>> > backpressure on the relationship to the processor after the RouteOnAttribute
>> > set to 1, but that queue was empty. There was no backpressure on the queue
>> > heading in to the RouteOnAttribute processor. The RouteOnAttribute processor
>> > has a Penalty Duration and a Yield of 1 sec. The relationships all used the
>> > PriorityAttributeAnalyzer as the prioritizer.
>> >
>> > The interesting thing is I had the exact same problem on two different nifis
>> > (one was version 0.2.0 and one 0.3.0). The same type of flow files were
>> > going from one to the other using site-to-site (I was doing this for load
>> > balancing). On the other Nifi, instead of restarting, I added another
>> > RouteOnAttribute processor and rerouted the stuck queue to the new one
>> > instead, and that also got things moving again. (I have a backpressure since
>> > the processor after the RouteOnAttribute is a ControlRate processor, and I
>> > only want to allow 1 flowfile through per second so as not to overwhelm the
>> > system downstream. The backpressure is to assist with load balancing, so if
>> > one pathway fills up, I have the other pathway which uses the site-to-site
>> > to route the flow to the other nifi).
>> >
>> > Is there some combination of backpressure, penalty or yield along with
>> > prioritization that could cause a kind of deadlock-like situation? Any ideas
>> > as to how I can prevent this from occurring?
>> >
>> > Thanks!
>> > -Elli
>> >
>>
>>
>>
>> --
>> Sincerely,
>> Chris Hamm
>> (E) cehamm01@gmail.com <ma...@gmail.com>
>> (Twitter) http://twitter.com/webhamm <http://twitter.com/webhamm>
>> (Linkedin) http://www.linkedin.com/in/chrishamm <http://www.linkedin.com/in/chrishamm>
>
Re: Queue wasn't emptying
Posted by Christopher Hamm <ce...@gmail.com>.
that isn't clearing them out really. it gradually decreases but doesn't
clear out 500k items.
On Nov 12, 2015 8:22 AM, "Juan Sequeiros" <ju...@onyxpoint.com>
wrote:
> To answer Christopher,
>
> You can clear out flowfiles on a queue by setting expiration on
> relationship line.
>
> On Thu, Nov 12, 2015 at 7:53 AM, Christopher Hamm <ce...@gmail.com>
> wrote:
>
>> I will look, but how do I just kill it with fire. What can I delete?
>> On Nov 12, 2015 7:42 AM, "Mark Payne" <ma...@hotmail.com> wrote:
>>
>>> Chris / Elli,
>>>
>>> What version of NiFi are you running? I am wondering if you may have
>>> been bitten by
>>> this bug [1]. This should affect only the 'master' branch, not any
>>> formally released
>>> versions of NiFi. If you built the application from source code, rather
>>> than using an
>>> already-released version, this may well be what is happening.
>>>
>>> Thanks
>>> -Mark
>>>
>>>
>>> [1] https://issues.apache.org/jira/browse/NIFI-1155
>>>
>>>
>>> On Nov 12, 2015, at 2:32 AM, Christopher Hamm <em...@christopherhamm.com>
>>> wrote:
>>>
>>> where does it dump? I cant get mine to expire fast enough. It is dev
>>> test I want to blow away but cant delete.
>>>
>>> On Fri, Nov 6, 2015 at 11:33 AM, Joe Witt <jo...@gmail.com> wrote:
>>>
>>>> Elli,
>>>>
>>>> Can you share your flow configuration that we could possibly use to
>>>> replicate this? Perhaps turn this into a JIRA and attach that.
>>>>
>>>> Also anytime you see something that appears 'stuck' please try to get
>>>> a stack dump (bin/nifi.sh dump) As if there is truly a stuck thread
>>>> we'll see it there and what it is blocked on.
>>>>
>>>> Thanks
>>>> Joe
>>>>
>>>> On Fri, Nov 6, 2015 at 11:25 AM, Elli Schwarz <
>>>> eliezer_schwarz@yahoo.com> wrote:
>>>> > I had a queue that built up overnight with several thousand
>>>> flowfiles. The
>>>> > queue was pointing to a RouteOnAttribute processor, which was
>>>> running. For
>>>> > some reason, the RouteOnAttribute processor wasn't emptying the queue
>>>> so the
>>>> > queue just built up. Stopping and starting the processor didn't help.
>>>> > However, simply restarting Nifi got the flow moving again. I can't
>>>> figure
>>>> > out why the queue was stuck.
>>>> >
>>>> > What could cause a queue to build up like that? How would restarting
>>>> Nifi
>>>> > get it going again? There weren't any errors in the log. I did have
>>>> > backpressure on the relationship to the processor after the
>>>> RouteOnAttribute
>>>> > set to 1, but that queue was empty. There was no backpressure on the
>>>> queue
>>>> > heading in to the RouteOnAttribute processor. The RouteOnAttribute
>>>> processor
>>>> > has a Penalty Duration and a Yield of 1 sec. The relationships all
>>>> used the
>>>> > PriorityAttributeAnalyzer as the prioritizer.
>>>> >
>>>> > The interesting thing is I had the exact same problem on two
>>>> different nifis
>>>> > (one was version 0.2.0 and one 0.3.0). The same type of flow files
>>>> were
>>>> > going from one to the other using site-to-site (I was doing this for
>>>> load
>>>> > balancing). On the other Nifi, instead of restarting, I added another
>>>> > RouteOnAttribute processor and rerouted the stuck queue to the new one
>>>> > instead, and that also got things moving again. (I have a
>>>> backpressure since
>>>> > the processor after the RouteOnAttribute is a ControlRate processor,
>>>> and I
>>>> > only want to allow 1 flowfile through per second so as not to
>>>> overwhelm the
>>>> > system downstream. The backpressure is to assist with load balancing,
>>>> so if
>>>> > one pathway fills up, I have the other pathway which uses the
>>>> site-to-site
>>>> > to route the flow to the other nifi).
>>>> >
>>>> > Is there some combination of backpressure, penalty or yield along with
>>>> > prioritization that could cause a kind of deadlock-like situation?
>>>> Any ideas
>>>> > as to how I can prevent this from occurring?
>>>> >
>>>> > Thanks!
>>>> > -Elli
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> Sincerely,
>>> Chris Hamm
>>> (E) cehamm01@gmail.com
>>> (Twitter) http://twitter.com/webhamm
>>> (Linkedin) http://www.linkedin.com/in/chrishamm
>>>
>>>
>>>
>
Re: Queue wasn't emptying
Posted by Juan Sequeiros <ju...@onyxpoint.com>.
To answer Christopher,
You can clear out flowfiles on a queue by setting expiration on
relationship line.
On Thu, Nov 12, 2015 at 7:53 AM, Christopher Hamm <ce...@gmail.com>
wrote:
> I will look, but how do I just kill it with fire. What can I delete?
> On Nov 12, 2015 7:42 AM, "Mark Payne" <ma...@hotmail.com> wrote:
>
>> Chris / Elli,
>>
>> What version of NiFi are you running? I am wondering if you may have been
>> bitten by
>> this bug [1]. This should affect only the 'master' branch, not any
>> formally released
>> versions of NiFi. If you built the application from source code, rather
>> than using an
>> already-released version, this may well be what is happening.
>>
>> Thanks
>> -Mark
>>
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-1155
>>
>>
>> On Nov 12, 2015, at 2:32 AM, Christopher Hamm <em...@christopherhamm.com>
>> wrote:
>>
>> where does it dump? I cant get mine to expire fast enough. It is dev test
>> I want to blow away but cant delete.
>>
>> On Fri, Nov 6, 2015 at 11:33 AM, Joe Witt <jo...@gmail.com> wrote:
>>
>>> Elli,
>>>
>>> Can you share your flow configuration that we could possibly use to
>>> replicate this? Perhaps turn this into a JIRA and attach that.
>>>
>>> Also anytime you see something that appears 'stuck' please try to get
>>> a stack dump (bin/nifi.sh dump) As if there is truly a stuck thread
>>> we'll see it there and what it is blocked on.
>>>
>>> Thanks
>>> Joe
>>>
>>> On Fri, Nov 6, 2015 at 11:25 AM, Elli Schwarz <el...@yahoo.com>
>>> wrote:
>>> > I had a queue that built up overnight with several thousand flowfiles.
>>> The
>>> > queue was pointing to a RouteOnAttribute processor, which was running.
>>> For
>>> > some reason, the RouteOnAttribute processor wasn't emptying the queue
>>> so the
>>> > queue just built up. Stopping and starting the processor didn't help.
>>> > However, simply restarting Nifi got the flow moving again. I can't
>>> figure
>>> > out why the queue was stuck.
>>> >
>>> > What could cause a queue to build up like that? How would restarting
>>> Nifi
>>> > get it going again? There weren't any errors in the log. I did have
>>> > backpressure on the relationship to the processor after the
>>> RouteOnAttribute
>>> > set to 1, but that queue was empty. There was no backpressure on the
>>> queue
>>> > heading in to the RouteOnAttribute processor. The RouteOnAttribute
>>> processor
>>> > has a Penalty Duration and a Yield of 1 sec. The relationships all
>>> used the
>>> > PriorityAttributeAnalyzer as the prioritizer.
>>> >
>>> > The interesting thing is I had the exact same problem on two different
>>> nifis
>>> > (one was version 0.2.0 and one 0.3.0). The same type of flow files were
>>> > going from one to the other using site-to-site (I was doing this for
>>> load
>>> > balancing). On the other Nifi, instead of restarting, I added another
>>> > RouteOnAttribute processor and rerouted the stuck queue to the new one
>>> > instead, and that also got things moving again. (I have a backpressure
>>> since
>>> > the processor after the RouteOnAttribute is a ControlRate processor,
>>> and I
>>> > only want to allow 1 flowfile through per second so as not to
>>> overwhelm the
>>> > system downstream. The backpressure is to assist with load balancing,
>>> so if
>>> > one pathway fills up, I have the other pathway which uses the
>>> site-to-site
>>> > to route the flow to the other nifi).
>>> >
>>> > Is there some combination of backpressure, penalty or yield along with
>>> > prioritization that could cause a kind of deadlock-like situation? Any
>>> ideas
>>> > as to how I can prevent this from occurring?
>>> >
>>> > Thanks!
>>> > -Elli
>>> >
>>>
>>
>>
>>
>> --
>> Sincerely,
>> Chris Hamm
>> (E) cehamm01@gmail.com
>> (Twitter) http://twitter.com/webhamm
>> (Linkedin) http://www.linkedin.com/in/chrishamm
>>
>>
>>
Re: Queue wasn't emptying
Posted by Mark Payne <ma...@hotmail.com>.
Chris,
Do you have Expiration set on the queue that RouteOnAttribute is feeding?
If so, and if this is in fact the bug that you're hitting, then you can remove expiration
on the connection to prevent this from happening and restart NiFi to make it resume.
Thanks
-Mark
> On Nov 12, 2015, at 7:53 AM, Christopher Hamm <ce...@gmail.com> wrote:
>
> I will look, but how do I just kill it with fire. What can I delete?
>
> On Nov 12, 2015 7:42 AM, "Mark Payne" <markap14@hotmail.com <ma...@hotmail.com>> wrote:
> Chris / Elli,
>
> What version of NiFi are you running? I am wondering if you may have been bitten by
> this bug [1]. This should affect only the 'master' branch, not any formally released
> versions of NiFi. If you built the application from source code, rather than using an
> already-released version, this may well be what is happening.
>
> Thanks
> -Mark
>
>
> [1] https://issues.apache.org/jira/browse/NIFI-1155 <https://issues.apache.org/jira/browse/NIFI-1155>
>
>
>> On Nov 12, 2015, at 2:32 AM, Christopher Hamm <email@christopherhamm.com <ma...@christopherhamm.com>> wrote:
>>
>> where does it dump? I cant get mine to expire fast enough. It is dev test I want to blow away but cant delete.
>>
>> On Fri, Nov 6, 2015 at 11:33 AM, Joe Witt <joe.witt@gmail.com <ma...@gmail.com>> wrote:
>> Elli,
>>
>> Can you share your flow configuration that we could possibly use to
>> replicate this? Perhaps turn this into a JIRA and attach that.
>>
>> Also anytime you see something that appears 'stuck' please try to get
>> a stack dump (bin/nifi.sh dump) As if there is truly a stuck thread
>> we'll see it there and what it is blocked on.
>>
>> Thanks
>> Joe
>>
>> On Fri, Nov 6, 2015 at 11:25 AM, Elli Schwarz <eliezer_schwarz@yahoo.com <ma...@yahoo.com>> wrote:
>> > I had a queue that built up overnight with several thousand flowfiles. The
>> > queue was pointing to a RouteOnAttribute processor, which was running. For
>> > some reason, the RouteOnAttribute processor wasn't emptying the queue so the
>> > queue just built up. Stopping and starting the processor didn't help.
>> > However, simply restarting Nifi got the flow moving again. I can't figure
>> > out why the queue was stuck.
>> >
>> > What could cause a queue to build up like that? How would restarting Nifi
>> > get it going again? There weren't any errors in the log. I did have
>> > backpressure on the relationship to the processor after the RouteOnAttribute
>> > set to 1, but that queue was empty. There was no backpressure on the queue
>> > heading in to the RouteOnAttribute processor. The RouteOnAttribute processor
>> > has a Penalty Duration and a Yield of 1 sec. The relationships all used the
>> > PriorityAttributeAnalyzer as the prioritizer.
>> >
>> > The interesting thing is I had the exact same problem on two different nifis
>> > (one was version 0.2.0 and one 0.3.0). The same type of flow files were
>> > going from one to the other using site-to-site (I was doing this for load
>> > balancing). On the other Nifi, instead of restarting, I added another
>> > RouteOnAttribute processor and rerouted the stuck queue to the new one
>> > instead, and that also got things moving again. (I have a backpressure since
>> > the processor after the RouteOnAttribute is a ControlRate processor, and I
>> > only want to allow 1 flowfile through per second so as not to overwhelm the
>> > system downstream. The backpressure is to assist with load balancing, so if
>> > one pathway fills up, I have the other pathway which uses the site-to-site
>> > to route the flow to the other nifi).
>> >
>> > Is there some combination of backpressure, penalty or yield along with
>> > prioritization that could cause a kind of deadlock-like situation? Any ideas
>> > as to how I can prevent this from occurring?
>> >
>> > Thanks!
>> > -Elli
>> >
>>
>>
>>
>> --
>> Sincerely,
>> Chris Hamm
>> (E) cehamm01@gmail.com <ma...@gmail.com>
>> (Twitter) http://twitter.com/webhamm <http://twitter.com/webhamm>
>> (Linkedin) http://www.linkedin.com/in/chrishamm <http://www.linkedin.com/in/chrishamm>
Re: Queue wasn't emptying
Posted by Christopher Hamm <ce...@gmail.com>.
I will look, but how do I just kill it with fire. What can I delete?
On Nov 12, 2015 7:42 AM, "Mark Payne" <ma...@hotmail.com> wrote:
> Chris / Elli,
>
> What version of NiFi are you running? I am wondering if you may have been
> bitten by
> this bug [1]. This should affect only the 'master' branch, not any
> formally released
> versions of NiFi. If you built the application from source code, rather
> than using an
> already-released version, this may well be what is happening.
>
> Thanks
> -Mark
>
>
> [1] https://issues.apache.org/jira/browse/NIFI-1155
>
>
> On Nov 12, 2015, at 2:32 AM, Christopher Hamm <em...@christopherhamm.com>
> wrote:
>
> where does it dump? I cant get mine to expire fast enough. It is dev test
> I want to blow away but cant delete.
>
> On Fri, Nov 6, 2015 at 11:33 AM, Joe Witt <jo...@gmail.com> wrote:
>
>> Elli,
>>
>> Can you share your flow configuration that we could possibly use to
>> replicate this? Perhaps turn this into a JIRA and attach that.
>>
>> Also anytime you see something that appears 'stuck' please try to get
>> a stack dump (bin/nifi.sh dump) As if there is truly a stuck thread
>> we'll see it there and what it is blocked on.
>>
>> Thanks
>> Joe
>>
>> On Fri, Nov 6, 2015 at 11:25 AM, Elli Schwarz <el...@yahoo.com>
>> wrote:
>> > I had a queue that built up overnight with several thousand flowfiles.
>> The
>> > queue was pointing to a RouteOnAttribute processor, which was running.
>> For
>> > some reason, the RouteOnAttribute processor wasn't emptying the queue
>> so the
>> > queue just built up. Stopping and starting the processor didn't help.
>> > However, simply restarting Nifi got the flow moving again. I can't
>> figure
>> > out why the queue was stuck.
>> >
>> > What could cause a queue to build up like that? How would restarting
>> Nifi
>> > get it going again? There weren't any errors in the log. I did have
>> > backpressure on the relationship to the processor after the
>> RouteOnAttribute
>> > set to 1, but that queue was empty. There was no backpressure on the
>> queue
>> > heading in to the RouteOnAttribute processor. The RouteOnAttribute
>> processor
>> > has a Penalty Duration and a Yield of 1 sec. The relationships all used
>> the
>> > PriorityAttributeAnalyzer as the prioritizer.
>> >
>> > The interesting thing is I had the exact same problem on two different
>> nifis
>> > (one was version 0.2.0 and one 0.3.0). The same type of flow files were
>> > going from one to the other using site-to-site (I was doing this for
>> load
>> > balancing). On the other Nifi, instead of restarting, I added another
>> > RouteOnAttribute processor and rerouted the stuck queue to the new one
>> > instead, and that also got things moving again. (I have a backpressure
>> since
>> > the processor after the RouteOnAttribute is a ControlRate processor,
>> and I
>> > only want to allow 1 flowfile through per second so as not to overwhelm
>> the
>> > system downstream. The backpressure is to assist with load balancing,
>> so if
>> > one pathway fills up, I have the other pathway which uses the
>> site-to-site
>> > to route the flow to the other nifi).
>> >
>> > Is there some combination of backpressure, penalty or yield along with
>> > prioritization that could cause a kind of deadlock-like situation? Any
>> ideas
>> > as to how I can prevent this from occurring?
>> >
>> > Thanks!
>> > -Elli
>> >
>>
>
>
>
> --
> Sincerely,
> Chris Hamm
> (E) cehamm01@gmail.com
> (Twitter) http://twitter.com/webhamm
> (Linkedin) http://www.linkedin.com/in/chrishamm
>
>
>
Re: Queue wasn't emptying
Posted by Mark Payne <ma...@hotmail.com>.
Chris / Elli,
What version of NiFi are you running? I am wondering if you may have been bitten by
this bug [1]. This should affect only the 'master' branch, not any formally released
versions of NiFi. If you built the application from source code, rather than using an
already-released version, this may well be what is happening.
Thanks
-Mark
[1] https://issues.apache.org/jira/browse/NIFI-1155 <https://issues.apache.org/jira/browse/NIFI-1155>
> On Nov 12, 2015, at 2:32 AM, Christopher Hamm <em...@christopherhamm.com> wrote:
>
> where does it dump? I cant get mine to expire fast enough. It is dev test I want to blow away but cant delete.
>
> On Fri, Nov 6, 2015 at 11:33 AM, Joe Witt <joe.witt@gmail.com <ma...@gmail.com>> wrote:
> Elli,
>
> Can you share your flow configuration that we could possibly use to
> replicate this? Perhaps turn this into a JIRA and attach that.
>
> Also anytime you see something that appears 'stuck' please try to get
> a stack dump (bin/nifi.sh dump) As if there is truly a stuck thread
> we'll see it there and what it is blocked on.
>
> Thanks
> Joe
>
> On Fri, Nov 6, 2015 at 11:25 AM, Elli Schwarz <eliezer_schwarz@yahoo.com <ma...@yahoo.com>> wrote:
> > I had a queue that built up overnight with several thousand flowfiles. The
> > queue was pointing to a RouteOnAttribute processor, which was running. For
> > some reason, the RouteOnAttribute processor wasn't emptying the queue so the
> > queue just built up. Stopping and starting the processor didn't help.
> > However, simply restarting Nifi got the flow moving again. I can't figure
> > out why the queue was stuck.
> >
> > What could cause a queue to build up like that? How would restarting Nifi
> > get it going again? There weren't any errors in the log. I did have
> > backpressure on the relationship to the processor after the RouteOnAttribute
> > set to 1, but that queue was empty. There was no backpressure on the queue
> > heading in to the RouteOnAttribute processor. The RouteOnAttribute processor
> > has a Penalty Duration and a Yield of 1 sec. The relationships all used the
> > PriorityAttributeAnalyzer as the prioritizer.
> >
> > The interesting thing is I had the exact same problem on two different nifis
> > (one was version 0.2.0 and one 0.3.0). The same type of flow files were
> > going from one to the other using site-to-site (I was doing this for load
> > balancing). On the other Nifi, instead of restarting, I added another
> > RouteOnAttribute processor and rerouted the stuck queue to the new one
> > instead, and that also got things moving again. (I have a backpressure since
> > the processor after the RouteOnAttribute is a ControlRate processor, and I
> > only want to allow 1 flowfile through per second so as not to overwhelm the
> > system downstream. The backpressure is to assist with load balancing, so if
> > one pathway fills up, I have the other pathway which uses the site-to-site
> > to route the flow to the other nifi).
> >
> > Is there some combination of backpressure, penalty or yield along with
> > prioritization that could cause a kind of deadlock-like situation? Any ideas
> > as to how I can prevent this from occurring?
> >
> > Thanks!
> > -Elli
> >
>
>
>
> --
> Sincerely,
> Chris Hamm
> (E) cehamm01@gmail.com <ma...@gmail.com>
> (Twitter) http://twitter.com/webhamm <http://twitter.com/webhamm>
> (Linkedin) http://www.linkedin.com/in/chrishamm <http://www.linkedin.com/in/chrishamm>
Re: Queue wasn't emptying
Posted by Christopher Hamm <em...@christopherhamm.com>.
where does it dump? I cant get mine to expire fast enough. It is dev test I
want to blow away but cant delete.
On Fri, Nov 6, 2015 at 11:33 AM, Joe Witt <jo...@gmail.com> wrote:
> Elli,
>
> Can you share your flow configuration that we could possibly use to
> replicate this? Perhaps turn this into a JIRA and attach that.
>
> Also anytime you see something that appears 'stuck' please try to get
> a stack dump (bin/nifi.sh dump) As if there is truly a stuck thread
> we'll see it there and what it is blocked on.
>
> Thanks
> Joe
>
> On Fri, Nov 6, 2015 at 11:25 AM, Elli Schwarz <el...@yahoo.com>
> wrote:
> > I had a queue that built up overnight with several thousand flowfiles.
> The
> > queue was pointing to a RouteOnAttribute processor, which was running.
> For
> > some reason, the RouteOnAttribute processor wasn't emptying the queue so
> the
> > queue just built up. Stopping and starting the processor didn't help.
> > However, simply restarting Nifi got the flow moving again. I can't figure
> > out why the queue was stuck.
> >
> > What could cause a queue to build up like that? How would restarting Nifi
> > get it going again? There weren't any errors in the log. I did have
> > backpressure on the relationship to the processor after the
> RouteOnAttribute
> > set to 1, but that queue was empty. There was no backpressure on the
> queue
> > heading in to the RouteOnAttribute processor. The RouteOnAttribute
> processor
> > has a Penalty Duration and a Yield of 1 sec. The relationships all used
> the
> > PriorityAttributeAnalyzer as the prioritizer.
> >
> > The interesting thing is I had the exact same problem on two different
> nifis
> > (one was version 0.2.0 and one 0.3.0). The same type of flow files were
> > going from one to the other using site-to-site (I was doing this for load
> > balancing). On the other Nifi, instead of restarting, I added another
> > RouteOnAttribute processor and rerouted the stuck queue to the new one
> > instead, and that also got things moving again. (I have a backpressure
> since
> > the processor after the RouteOnAttribute is a ControlRate processor, and
> I
> > only want to allow 1 flowfile through per second so as not to overwhelm
> the
> > system downstream. The backpressure is to assist with load balancing, so
> if
> > one pathway fills up, I have the other pathway which uses the
> site-to-site
> > to route the flow to the other nifi).
> >
> > Is there some combination of backpressure, penalty or yield along with
> > prioritization that could cause a kind of deadlock-like situation? Any
> ideas
> > as to how I can prevent this from occurring?
> >
> > Thanks!
> > -Elli
> >
>
--
Sincerely,
Chris Hamm
(E) cehamm01@gmail.com
(Twitter) http://twitter.com/webhamm
(Linkedin) http://www.linkedin.com/in/chrishamm
Re: Queue wasn't emptying
Posted by Joe Witt <jo...@gmail.com>.
Elli,
Can you share your flow configuration that we could possibly use to
replicate this? Perhaps turn this into a JIRA and attach that.
Also anytime you see something that appears 'stuck' please try to get
a stack dump (bin/nifi.sh dump) As if there is truly a stuck thread
we'll see it there and what it is blocked on.
Thanks
Joe
On Fri, Nov 6, 2015 at 11:25 AM, Elli Schwarz <el...@yahoo.com> wrote:
> I had a queue that built up overnight with several thousand flowfiles. The
> queue was pointing to a RouteOnAttribute processor, which was running. For
> some reason, the RouteOnAttribute processor wasn't emptying the queue so the
> queue just built up. Stopping and starting the processor didn't help.
> However, simply restarting Nifi got the flow moving again. I can't figure
> out why the queue was stuck.
>
> What could cause a queue to build up like that? How would restarting Nifi
> get it going again? There weren't any errors in the log. I did have
> backpressure on the relationship to the processor after the RouteOnAttribute
> set to 1, but that queue was empty. There was no backpressure on the queue
> heading in to the RouteOnAttribute processor. The RouteOnAttribute processor
> has a Penalty Duration and a Yield of 1 sec. The relationships all used the
> PriorityAttributeAnalyzer as the prioritizer.
>
> The interesting thing is I had the exact same problem on two different nifis
> (one was version 0.2.0 and one 0.3.0). The same type of flow files were
> going from one to the other using site-to-site (I was doing this for load
> balancing). On the other Nifi, instead of restarting, I added another
> RouteOnAttribute processor and rerouted the stuck queue to the new one
> instead, and that also got things moving again. (I have a backpressure since
> the processor after the RouteOnAttribute is a ControlRate processor, and I
> only want to allow 1 flowfile through per second so as not to overwhelm the
> system downstream. The backpressure is to assist with load balancing, so if
> one pathway fills up, I have the other pathway which uses the site-to-site
> to route the flow to the other nifi).
>
> Is there some combination of backpressure, penalty or yield along with
> prioritization that could cause a kind of deadlock-like situation? Any ideas
> as to how I can prevent this from occurring?
>
> Thanks!
> -Elli
>