You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Joe Gresock <jg...@gmail.com> on 2016/09/29 16:02:02 UTC
Queue prioritization with backpressure
In my flow, I receive flow files from multiple sources that eventually get
funneled into one processor. The connections use OldestFirstPrioritizer,
and while there is no backpressure, the cross-queue prioritization appears
to work as designed (i.e., each queue that leads into the funnel appears to
be fairly drained, resulting in essentially a round robin distribution
among the funneled queues).
However, as soon as backpressure engages, NiFi appears to want to fully
drain one of the funneled queues, leaving the others untouched. This
happens even when the undrained queues have flow files older than the ones
that are drained first. Is this known behavior in NiFi? If so, do you
guys think it should be addressed?
--
I know what it is to be in need, and I know what it is to have plenty. I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want. I can do
all this through him who gives me strength. *-Philippians 4:12-13*
Re: Queue prioritization with backpressure
Posted by Joe Gresock <jg...@gmail.com>.
Thanks Pierre, the ticket is https://issues.apache.org/jira/browse/NIFI-2848
.
On Fri, Sep 30, 2016 at 11:29 AM, Pierre Villard <
pierre.villard.fr@gmail.com> wrote:
> Hi Joe,
>
> This, indeed, looks like something we should fix.
> Please create a JIRA and I'll try to have a look into it.
>
> Pierre
>
> 2016-09-30 12:01 GMT+02:00 Joe Gresock <jg...@gmail.com>:
>
> > I forgot to mention the bigger problem with the way this prioritization
> > works. If three queues lead to a single component, and all 3 are full
> due
> > to back pressure, it is easily observable that once Queue A starts to
> > drain, it will continue to drain as long as it has incoming flow files.
> > This means that if there's a constant flow of incoming flow files to
> Queue
> > A, the other two queues will never be drained (at least, that's my theory
> > based on an hour of observation).
> >
> > I'm going to create a JIRA ticket to track this.
> >
> > On Thu, Sep 29, 2016 at 4:02 PM, Joe Gresock <jg...@gmail.com> wrote:
> >
> > > In my flow, I receive flow files from multiple sources that eventually
> > get
> > > funneled into one processor. The connections use
> OldestFirstPrioritizer,
> > > and while there is no backpressure, the cross-queue prioritization
> > appears
> > > to work as designed (i.e., each queue that leads into the funnel
> appears
> > to
> > > be fairly drained, resulting in essentially a round robin distribution
> > > among the funneled queues).
> > >
> > > However, as soon as backpressure engages, NiFi appears to want to fully
> > > drain one of the funneled queues, leaving the others untouched. This
> > > happens even when the undrained queues have flow files older than the
> > ones
> > > that are drained first. Is this known behavior in NiFi? If so, do you
> > > guys think it should be addressed?
> > >
> > > --
> > > I know what it is to be in need, and I know what it is to have
> plenty. I
> > > have learned the secret of being content in any and every situation,
> > > whether well fed or hungry, whether living in plenty or in want. I can
> > > do all this through him who gives me strength. *-Philippians
> 4:12-13*
> > >
> >
> >
> >
> > --
> > I know what it is to be in need, and I know what it is to have plenty. I
> > have learned the secret of being content in any and every situation,
> > whether well fed or hungry, whether living in plenty or in want. I can
> do
> > all this through him who gives me strength. *-Philippians 4:12-13*
> >
>
--
I know what it is to be in need, and I know what it is to have plenty. I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want. I can do
all this through him who gives me strength. *-Philippians 4:12-13*
Re: Queue prioritization with backpressure
Posted by Pierre Villard <pi...@gmail.com>.
Hi Joe,
This, indeed, looks like something we should fix.
Please create a JIRA and I'll try to have a look into it.
Pierre
2016-09-30 12:01 GMT+02:00 Joe Gresock <jg...@gmail.com>:
> I forgot to mention the bigger problem with the way this prioritization
> works. If three queues lead to a single component, and all 3 are full due
> to back pressure, it is easily observable that once Queue A starts to
> drain, it will continue to drain as long as it has incoming flow files.
> This means that if there's a constant flow of incoming flow files to Queue
> A, the other two queues will never be drained (at least, that's my theory
> based on an hour of observation).
>
> I'm going to create a JIRA ticket to track this.
>
> On Thu, Sep 29, 2016 at 4:02 PM, Joe Gresock <jg...@gmail.com> wrote:
>
> > In my flow, I receive flow files from multiple sources that eventually
> get
> > funneled into one processor. The connections use OldestFirstPrioritizer,
> > and while there is no backpressure, the cross-queue prioritization
> appears
> > to work as designed (i.e., each queue that leads into the funnel appears
> to
> > be fairly drained, resulting in essentially a round robin distribution
> > among the funneled queues).
> >
> > However, as soon as backpressure engages, NiFi appears to want to fully
> > drain one of the funneled queues, leaving the others untouched. This
> > happens even when the undrained queues have flow files older than the
> ones
> > that are drained first. Is this known behavior in NiFi? If so, do you
> > guys think it should be addressed?
> >
> > --
> > I know what it is to be in need, and I know what it is to have plenty. I
> > have learned the secret of being content in any and every situation,
> > whether well fed or hungry, whether living in plenty or in want. I can
> > do all this through him who gives me strength. *-Philippians 4:12-13*
> >
>
>
>
> --
> I know what it is to be in need, and I know what it is to have plenty. I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want. I can do
> all this through him who gives me strength. *-Philippians 4:12-13*
>
Re: Queue prioritization with backpressure
Posted by Joe Gresock <jg...@gmail.com>.
I forgot to mention the bigger problem with the way this prioritization
works. If three queues lead to a single component, and all 3 are full due
to back pressure, it is easily observable that once Queue A starts to
drain, it will continue to drain as long as it has incoming flow files.
This means that if there's a constant flow of incoming flow files to Queue
A, the other two queues will never be drained (at least, that's my theory
based on an hour of observation).
I'm going to create a JIRA ticket to track this.
On Thu, Sep 29, 2016 at 4:02 PM, Joe Gresock <jg...@gmail.com> wrote:
> In my flow, I receive flow files from multiple sources that eventually get
> funneled into one processor. The connections use OldestFirstPrioritizer,
> and while there is no backpressure, the cross-queue prioritization appears
> to work as designed (i.e., each queue that leads into the funnel appears to
> be fairly drained, resulting in essentially a round robin distribution
> among the funneled queues).
>
> However, as soon as backpressure engages, NiFi appears to want to fully
> drain one of the funneled queues, leaving the others untouched. This
> happens even when the undrained queues have flow files older than the ones
> that are drained first. Is this known behavior in NiFi? If so, do you
> guys think it should be addressed?
>
> --
> I know what it is to be in need, and I know what it is to have plenty. I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want. I can
> do all this through him who gives me strength. *-Philippians 4:12-13*
>
--
I know what it is to be in need, and I know what it is to have plenty. I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want. I can do
all this through him who gives me strength. *-Philippians 4:12-13*